Trouble compiling pycaffe, "Python.h : No such file or directory..."

[michae...@brown.edu]

Hi! I'm very new to caffe as well as to coding in Python so any advice on how to correct the following error would be greatly appreciated!

my make file is:

CUSTOM_CXX := g++

CUD_DIR:= /usr/local/cuda

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \

-gencode arch=compute_20,code=sm_21 \

-gencode arch=compute_30,code=sm_30 \

-gencode arch=compute_35,code=sm_35 \

-gencode arch=compute_50,code=sm_50 \

-gencode arch=compute_50,code=compute_50

BLAS := mkl

PYTHON_INCLUDE := /usr/include/python2.7 \

/usr/lib/python2.7/dist-packages/numpy/core/include/

PYTHON_LIB := /usr/lib

BUILD_DIR := build

DISTRIBUTE_DIR :=distribute

TEST_GPUID :=0

Q ?= @

I'm running Ubuntu 14.04 with a GeForce GTX660. I can run make all, make test, and make runtest just fine with no errors. However when I then run make pycaffe I get the following error:

python/caffe/_caffe.cpp:1:52: fatal error: Python.h: no such file or directory

 #include <Python.h> // NOLINT(build/include_alpha)

compilation terminated.

make: *** [python/caffe/_caffe.so] Error 1

Any suggestions on how to fix this pointer error would be greatly appreciated!

Thanks,

Max

[Steven]

You might want to try:

sudo apt-get install python-dev

in order to install your header files.

As a general rule of thumb, whenever you get an error you don't understand, try Googling the error message. Frequently, others have had the same problem. StackOverflow is also a good site to ask questions and search for answers on.

[michae...@brown.edu]

Hi Steven,

Thanks for the quick response! I have definitely installed python 2.7 and executed apt-get install python-dev. So my header files are there. I assume that Python.h indicates a missing header file however, when I look in /usr/include/python2.7 (which is part of PYTHON_INCLUDE in my make file) Python.h is most definitely there...

Any idea why the caffe compiler would be missing this?

Thanks again for the help!

-Max

[Steven]

Hi Max,

I'm afraid I don't really know what to say. As a next step, I might try looking at which compilation caused the error.

I just tried 'sudo make pycaffe' and got the following output:

$ sudo make pycaffe
[sudo] password for me:
NVCC src/caffe/layers/silence_layer.cu
NVCC src/caffe/layers/deconv_layer.cu
NVCC src/caffe/layers/tanh_layer.cu
NVCC src/caffe/layers/pooling_layer.cu
NVCC src/caffe/layers/mvn_layer.cu
NVCC src/caffe/layers/sigmoid_cross_entropy_loss_layer.cu
NVCC src/caffe/layers/concat_layer.cu
NVCC src/caffe/layers/dropout_layer.cu
NVCC src/caffe/layers/sigmoid_layer.cu
NVCC src/caffe/layers/contrastive_loss_layer.cu
NVCC src/caffe/layers/base_data_layer.cu
NVCC src/caffe/layers/cudnn_pooling_layer.cu
NVCC src/caffe/layers/bnll_layer.cu
NVCC src/caffe/layers/split_layer.cu
NVCC src/caffe/layers/im2col_layer.cu
NVCC src/caffe/layers/slice_layer.cu
NVCC src/caffe/layers/hdf5_data_layer.cu
NVCC src/caffe/layers/hdf5_output_layer.cu
NVCC src/caffe/layers/relu_layer.cu
NVCC src/caffe/layers/eltwise_layer.cu
NVCC src/caffe/layers/cudnn_softmax_layer.cu
NVCC src/caffe/layers/cudnn_relu_layer.cu
NVCC src/caffe/layers/cudnn_sigmoid_layer.cu
NVCC src/caffe/layers/lrn_layer.cu
NVCC src/caffe/layers/prelu_layer.cu
NVCC src/caffe/layers/threshold_layer.cu
NVCC src/caffe/layers/exp_layer.cu
NVCC src/caffe/layers/inner_product_layer.cu
NVCC src/caffe/layers/softmax_loss_layer.cu
NVCC src/caffe/layers/absval_layer.cu
NVCC src/caffe/layers/cudnn_tanh_layer.cu
NVCC src/caffe/layers/cudnn_conv_layer.cu
NVCC src/caffe/layers/power_layer.cu
NVCC src/caffe/layers/softmax_layer.cu
NVCC src/caffe/layers/euclidean_loss_layer.cu
NVCC src/caffe/layers/conv_layer.cu
NVCC src/caffe/util/im2col.cu
NVCC src/caffe/util/math_functions.cu
LD -o .build_release/lib/libcaffe.so
me@ARL-M6800:~/Caffe/caffe$

So, I think things worked for me.

My only other guesses for you are to wonder if you're using Anaconda python, which according to the docs, requires some modifications, or if there's something subtly off with your makefile.config flie (I'm assuming that what you showed as your 'Makefile' is really an abridged version of your Makefile.config.). For what it's worth, here's mine:

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
        -gencode arch=compute_20,code=sm_21 \
        -gencode arch=compute_30,code=sm_30 \
        -gencode arch=compute_35,code=sm_35 \
        -gencode arch=compute_50,code=sm_50 \
        -gencode arch=compute_50,code=compute_50

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
        /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
        # $(ANACONDA_HOME)/include/python2.7 \
        # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

Sorry I couldn't be of more help. When I first saw your post, it looked like one of the more 'standard' mistakes.


Good luck!

[michae...@brown.edu]

Steven,

Thanks so much for the advice. I totally uninstalled Anaconda and rebuilt my python libraries and everything seems to be compiling fine now! Thanks for the tip about people having issues with Anaconda.

Best,

Max

[Nghiem Quan Vinh Anh]

A bit late for the party but FYI I was able to fix this issue by uncommenting these lines:

# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
        # $(ANACONDA_HOME)/include/python2.7 \
        # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

Somehow I forgot to uncomment line 3 & 4.

[Wesley Hsieh]

I had this exact problem, except I fixed it by adding to the "PYTHON_INCLUDE" variable in the Makefile.config file.

I added the location of the "Python.h" file (/usr/include/python2.7/):

PYTHON_INCLUDE := /usr/local/lib/python2.7/dist-packages/numpy/core/include /usr/include/python2.7/

[Gray Embry]

Hi!

I was having this exact same problem.

I just managed to fix it by changing this section in Makefile.config:

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.

#PYTHON_INCLUDE := /usr/include/python2.7 \

 /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.

ANACONDA_HOME := $(HOME)/anaconda
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
 $(ANACONDA_HOME)/include/python2.7 \
 $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
            $(ANACONDA_HOME)

To this:

# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

#PYTHON_INCLUDE := /usr/include/python2.7 \

/usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

ANACONDA_HOME := $(HOME)/anaconda

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

$(ANACONDA_HOME)/include/python2.7 \

$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

     /usr/local/somewhere/home/embry/anaconda/include \

     /usr/local/somewhere/home/embry/anaconda/include/python2.7 \

            $(ANACONDA_HOME)

To figure this out, I'd commented out the final line in the file

# enable pretty build (comment to see full commands)

#Q ?= @

which let me see where the -I info, and from that I realized even though $ANACONDA_HOME was set in the config file, for some reason it was not passing the actual path I knew Python.h to be in.

Adding a full path to Anaconda's /include and /include/python2.7 directories enabled this to work (I imagine only one of them is entirely necessary).

--Gray

[Gray Embry]

Gah, just realized Wesley basically said the same thing I did, in fewer words :)

Oh well, perhaps the added detail will help someone.

--Gray

[Tim Holme]

This fixed it for me

[JP]

Thanks for the tip!

My anaconda installation was in /opt/anaconda2 (to make available to all users on the server), so I had to modify my make file as such:

ANACONDA_HOME := /opt/anaconda2 # instead of $(HOME)/anaconda

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

                $(ANACONDA_HOME)/include/python2.7 \

                $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

[Ladislav Jech]

I wasn't able to get it working on my Gentoo system and didn't want to stuck to Anaconda binary copy as non-gentoo way, something must be broken, not sure what, but I upgraded to 3.4 (currently stable branch in Gentoo ecosystem) and after some changes I was able to compile, details here;

https://stackoverflow.com/questions/39874099/caffegpuopencv3-1python3-5anacondafatal-error-python-h-no-such-file-or-di/44135622#44135622

After this single change to switch from 2 to 3 it works again.

Regards,

Ladislav

Dne čtvrtek 14. července 2016 8:14:50 UTC+2 JP napsal(a):