hdf5.h not found when building on ubuntu 15.04

[Jean-Patrick Pommier]

Hello

After having followed http://caffe.berkeleyvision.org/installation.html#compilation, make all failed with the following error:

~/App/caffe$ make all

PROTOC src/caffe/proto/caffe.proto
CXX .build_release/src/caffe/proto/caffe.pb.cc
CXX src/caffe/layer_factory.cpp
In file included from ./include/caffe/common_layers.hpp:10:0,
 from ./include/caffe/vision_layers.hpp:10,
 from src/caffe/layer_factory.cpp:6:
./include/caffe/data_layers.hpp:9:18: fatal error: hdf5.h: Aucun fichier ou dossier de ce type
 #include "hdf5.h"
 ^
compilation terminated.
Makefile:512: recipe for target '.build_release/src/caffe/layer_factory.o' failed
make: *** [.build_release/src/caffe/layer_factory.o] Error 1

Hdf5 files were first installed from repo with apt-get. I also build hdf5 from source with ./confifure make make install.hdf5.h itself can be found:

$ locate hdf5.h
/home/jeanpat/App/OPENCVsource/opencv/modules/flann/include/opencv2/flann/hdf5.h
/usr/include/hdf5/serial/hdf5.h
/usr/include/opencv2/flann/hdf5.h
/usr/include/vtk-6.1/vtk_hdf5.h
/usr/local/include/opencv2/flann/hdf5.h

Modifying the path to hdf5.h in data_layers.hpp didn't fix the problem:

#include "/usr/include/hdf5/serial/hdf5.h"
//"hdf5.h"
//"/usr/include/hdf5"

thanks for help

[Mohit Jain]

Hi,
    Fixing the hdf5.h path in the .cpp file is not a good fix in general. I'd suggest you install the dependencies provided in the python/requirements.txt file 

pip install -r /path/to/caffe/python/requirements.txt

Also, make sure you have the following. (apt-get)




glog, gflags, protobuf, leveldb, snappy, hdf5, lmdb

Then follow a similar pattern as shown below (im a mac user, so please check your paths accordingly)

The right tweaks in Makefile.config for me were : (might not be the most optimised caffe installation, but sure does work)

[dont use Anaconda]
: set CPU_ONLY:=1
: unset USE_CUDNN (simply comment it out - "#USE_CUDNN:=1")
: set BLAS:=(to the library you have installed) (I use a macbook and used atlas - default)
: set PYTHON_INCLUDE := /usr/include/python2.7 \

        /usr/local/lib/python2.7/site-packages/numpy/core/include/    
(paths might be different for specific users - but for noobs (like me) - this is where it should be by default) (check where do you have your python installed)

Hope this helps you. I've tested this on my MacBook Pro (OSX 10.10.3). Works smoothly via manual makefile.config edits as well cmake (tested both).

-Mohit

[Jean-Patrick Pommier]

Thanks for your attention,
I do read the ubuntu requirements http://caffe.berkeleyvision.org/install_apt.html

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev

and:

sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev protobuf-compiler

I have rechecked according to your advice:
sudo pip install -r /path/to/caffe/python/requirements.txt

all was installed


Regarding Makefile.config, I have:

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (comment to build without cuDNN).
#USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
 CPU_ONLY := 1

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
#CUDA_DIR := /usr/local/cuda

# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
 CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
 -gencode arch=compute_20,code=sm_21 \
 -gencode arch=compute_30,code=sm_30 \
 -gencode arch=compute_35,code=sm_35 \
 -gencode arch=compute_50,code=sm_50 \
 -gencode arch=compute_50,code=compute_50

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
 /usr/lib/python2.7/dist-packages/numpy/core/include \
 /usr/local/lib/python2.7/dist-packages/numpy/core/include

[Mohit Jain]

Are you sure that your CUDA_DIR is correctly set?
 CUDA_DIR := /usr (I believe this should be something like - CUDA_DIR := /usr/local/cuda) (or where ever you have cuda libs installed)

-Mohit

On Thursday, May 28, 2015 at 3:37:38 PM UTC+5:30, Jean-Patrick Pommier wrote:

[Jean-Patrick Pommier]

you're right, CUDA is in /usr/local but that doesn't fix the problem:

$ make all
CXX src/caffe/layer_factory.cpp

In file included from ./include/caffe/common_layers.hpp:10:0,
                 from ./include/caffe/vision_layers.hpp:10,
                 from src/caffe/layer_factory.cpp:6:

./include/caffe/data_layers.hpp:11:18: fatal error: hdf5.h: Aucun fichier ou dossier de ce type
 #include "hdf5.h"

                  ^
compilation terminated.
Makefile:512: recipe for target '.build_release/src/caffe/layer_factory.o' failed
make: *** [.build_release/src/caffe/layer_factory.o] Error 1

Le jeudi 28 mai 2015 12:07:38 UTC+2, Jean-Patrick Pommier a écrit :

[Edward Grant]

Identical problem with Lubuntu 15.04.

[Christopher Tensmeyer]

Hi, your problem is that Ubuntu 15 isn't sticking the HDF5 library where your compiler is currently looking for it.  It is not a problem with either CUDA or python.

I solved this issue and was able to get a clean build on a fresh Ubuntu 15 system.  Just install the dependencies as you did and then let the compiler know where to find them.

Specifically on Ubuntu 15, the HDF5 libraries and headers were installed in buried folders: respectively /usr/lib/x86_64-linux-gnu/hdf5/serial/ and /usr/include/hdf5/serial/

Create the following sym links:  

/usr/lib/x86_64-linux-gnu/hdf5/serial/hdf5_hl.so  ->  /usr/lib/x86_64-linux-gnu/hdf5_hl.so

/usr/lib/x86_64-linux-gnu/hdf5/serial/hdf5_hl.a  ->  /usr/lib/x86_64-linux-gnu/hdf5_hl.a

/usr/lib/x86_64-linux-gnu/hdf5/serial/hdf5.so  ->  /usr/lib/x86_64-linux-gnu/hdf5.so

/usr/lib/x86_64-linux-gnu/hdf5/serial/hdf5.a  ->  /usr/lib/x86_64-linux-gnu/hdf5.a

Many of the other hdf5 libs already had such sym links.  No idea why these specific libs were excluded.

Now your compiler is complaining about not finding the hdf5.h include file.  To solve this, I added the following line to the Makefile (though there are certain to be better solutions out there):

INCLUDE_DIRS += /usr/include/hdf5/serial/

Put is before this line: COMMON_FLAGS += $(foreach includedir,$(INCLUDE_DIRS),-I$(includedir))

Double check all of the paths because your config may be different than mine.

[Jean-Patrick Pommier]

Thanks a lot, with your indications the compilation can start. However some problems remain:

/usr/bin/ld: ne peut trouver -lhdf5_hl
/usr/bin/ld: ne peut trouver -lhdf5
collect2: error: ld returned 1 exit status
Makefile:506: recipe for target '.build_release/lib/libcaffe-nv.so.0.11.0' failed
make: *** [.build_release/lib/libcaffe-nv.so.0.11.0] Error 1

The hdf5 files are here:

:/usr/lib/x86_64-linux-gnu/hdf5/serial$ ls -la
total 8
drwxr-xr-x 2 root root 4096 mai   30 15:40 .
drwxr-xr-x 5 root root 4096 mai   30 16:02 ..
lrwxrwxrwx 1 root root   31 nov.   3  2014 include -> ../../../../include/hdf5/serial
lrwxrwxrwx 1 root root    1 nov.   3  2014 lib -> .
lrwxrwxrwx 1 root root   22 nov.   3  2014 libhdf5.a -> ../../libhdf5_serial.a
lrwxrwxrwx 1 root root   19 nov.   3  2014 libhdf5_cpp.a -> ../../libhdf5_cpp.a
lrwxrwxrwx 1 root root   20 nov.   3  2014 libhdf5_cpp.so -> ../../libhdf5_cpp.so
lrwxrwxrwx 1 root root   30 nov.   3  2014 libhdf5_fortran.a -> ../../libhdf5_serial_fortran.a
lrwxrwxrwx 1 root root   31 nov.   3  2014 libhdf5_fortran.so -> ../../libhdf5_serial_fortran.so
lrwxrwxrwx 1 root root   25 nov.   3  2014 libhdf5_hl.a -> ../../libhdf5_serial_hl.a
lrwxrwxrwx 1 root root   22 nov.   3  2014 libhdf5_hl_cpp.a -> ../../libhdf5_hl_cpp.a
lrwxrwxrwx 1 root root   23 nov.   3  2014 libhdf5_hl_cpp.so -> ../../libhdf5_hl_cpp.so
lrwxrwxrwx 1 root root   32 nov.   3  2014 libhdf5hl_fortran.a -> ../../libhdf5_serialhl_fortran.a
lrwxrwxrwx 1 root root   33 nov.   3  2014 libhdf5hl_fortran.so -> ../../libhdf5_serialhl_fortran.so
lrwxrwxrwx 1 root root   26 nov.   3  2014 libhdf5_hl.so -> ../../libhdf5_serial_hl.so
lrwxrwxrwx 1 root root   29 nov.   3  2014 libhdf5.settings -> ../../libhdf5_serial.settings
lrwxrwxrwx 1 root root   23 nov.   3  2014 libhdf5.so -> ../../libhdf5_serial.so

and the symbolic links are here:

jeanpat@jeanpat-WA50SHQ:/usr/lib/x86_64-linux-gnu/hdf5$ ls -la
total 288
drwxr-xr-x   5 root root   4096 mai   30 16:02 .
drwxr-xr-x 145 root root 274432 mai   28 22:16 ..
lrwxrwxrwx   1 root root     44 mai   30 16:02 hdf5.a -> /usr/lib/x86_64-linux-gnu/hdf5/serial/hdf5.a
lrwxrwxrwx   1 root root     47 mai   30 16:00 hdf5_hl.a -> /usr/lib/x86_64-linux-gnu/hdf5/serial/hdf5_hl.a
lrwxrwxrwx   1 root root     48 mai   30 16:01 hdf5_hl.so -> /usr/lib/x86_64-linux-gnu/hdf5/serial/hdf5_hl.so
lrwxrwxrwx   1 root root     19 mai   30 15:53 hdf5.so -> ./serial/libhdf5.so
drwxr-xr-x   2 root root   4096 mai   28 10:37 mpich
drwxr-xr-x   2 root root   4096 mai   28 10:37 openmpi
drwxr-xr-x   2 root root   4096 mai   30 15:40 serial

I am not sure that the problem is identical to the hdf5. issue.

[Christopher Tensmeyer]

Okay, it looks like its not finding the hdf5 library at link time.  Your setup looks a bit different because when I installed hdf5 there were many other sym links automatically created in the x86_64-linux-gnu folder.

What you need to do is run `ld --verbose -lhdf5` and that will tell you every folder the linker is looking for the hdf5 library.  Then make a sym link to one of those places (easy way), or google around to find out how to tell the linker to look in the x86_64 folder (not too much harder).  That should do it.

***Other Way***

Instead of modifying the MakeFile itself to find the include dir for hdf5/serial, modify the `INCLUDE_DIRS := ...` in Makefile.config.  You can also modify the `LIBRARY_DIRS := ...` line to point to where Ubuntu 15 stuck the hdf5 lib.  This way you can avoid making sym links or modifying the actual Makefile.  However, if you have other applications that use the hdf5 lib, it should get put somewhere LD will find it by default. 

...

[Coen Stevens]

I ran into the same issue. As suggested, changing the Makefile.config solves the problem. Check out https://github.com/BVLC/caffe/issues/2690 for more details.