compile caffe with python interface on macbook

[Xi Yin]

Hi there, I have a problem and I really could not figure it out by myself. Please help. Thank you. 

I try to compile caffe on the macbook. Here is some info for my macbook.

Processor  2.9 GHz Intel Core i7

Memory  8 GB 1600 MHz DDR3

Graphics  Intel HD Graphics 4000 1024 MB

Serial Number  C1MKXE96DTY4

Software  OS X 10.9.5 (13F1134)

It seems that the above GPU does not support CUDA. So I tried to compile with CPU only. I know the hardware may not be enough. I will have a more powerful setting desktop soon. Right now I just want to have some hands on experience to use Caffe. 

I installed all dependencies that are required. I also installed Anaconda Python under the path: /Users/xiyin/anaconda

I was able to run the following compiling commands:

make all

make test

make runtest

However when I tried: make pycaffe. It gave me the following error. 

It makes me so confused since it seemed there are multiple versions of pythons installed on my laptop and I am not sure if they have conflict somehow. There should be sth wrong with the Makefile.config file. For your reference, I have attached the file in the end. Thank you very much for those who could share their experience in compiling. 

====================

make: /Applications/MATLAB_R2012b.app/bin/mexext: Command not found

CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp

python/caffe/_caffe.cpp:1:10: fatal error: 'Python.h' file not found

#include <Python.h>  // NOLINT(build/include_alpha)

         ^

1 error generated.

make: *** [python/caffe/_caffe.so] Error 1

========================

Here is my Makefile.config:

======================

## Refer to http://caffe.berkeleyvision.org/installation.html

# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).

# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).

 CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers

# USE_OPENCV := 0

# USE_LEVELDB := 0

# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)

#       You should not set this flag if you will be reading LMDBs with any

#       possibility of simultaneous read and write

# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3

# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.

# N.B. the default for Linux is g++ and the default for OSX is clang++

# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.

CUDA_DIR := /usr/local/cuda

# On Ubuntu 14.04, if cuda tools are installed via

# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:

# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.

# For CUDA < 6.0, comment the *_50 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \

                -gencode arch=compute_20,code=sm_21 \

                -gencode arch=compute_30,code=sm_30 \

                -gencode arch=compute_35,code=sm_35 \

                -gencode arch=compute_50,code=sm_50 \

                -gencode arch=compute_50,code=compute_50

# BLAS choice:

# atlas for ATLAS (default)

# mkl for MKL

# open for OpenBlas

BLAS := atlas

# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.

# Leave commented to accept the defaults for your choice of BLAS

# (which should work)!

# BLAS_INCLUDE := /path/to/your/blas

# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path

# BLAS_INCLUDE := $(shell brew --prefix openblas)/include

# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.

# MATLAB directory should contain the mex binary in /bin.

# MATLAB_DIR := /usr/local

 MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

PYTHON_INCLUDE := /usr/include/python2.7 \

                /usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

 ANACONDA_HOME := $/Users/xiyin/anaconda

 PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

                # $(ANACONDA_HOME)/include/python2.7 \

                # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.

# PYTHON_LIB := /usr/lib

 PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)

# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include

# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)

# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies

# INCLUDE_DIRS += $(shell brew --prefix)/include

# LIBRARY_DIRS += $(shell brew --prefix)/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.

# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)

# USE_PKG_CONFIG := 1

BUILD_DIR := build

DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171

# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.

TEST_GPUID := 0

# enable pretty build (comment to see full commands)

Q ?= @

======================

[Jan C Peters]

Everything you said seems to be correct (yes for CUDA you need an NVidia GPU).

In your Makefile.config, try the following:
- comment out or remove the PYTHON_INCLUDE definition above the anaconda block
- in ANACONDA_HOME there seems to be $ that is not supposed to be there, just write /Users/xiyin/anaconda

Try to recompile and report results. The paths that are set there should be correct. You did install Anaconda with python 2.7, didn't you? Because pycaffe probably does not work with python 3.x.

[Xi Yin]

Hi Jan, 

Thank you so much for your quick reply. 

I changed the two points that you mentioned.  You have raised up a good question, I did install the Anaconda with python 3.4. So I uninstalled it and installed Anaconda with python 2.7 instead. I also uncomment the two lines that I used to comment before. This is what in the config file now. 

 ANACONDA_HOME := /Users/xiyin/anaconda

 PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

                 $(ANACONDA_HOME)/include/python2.7 \

                 $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

Here is the new error that I am getting. 

make: /Applications/MATLAB_R2012b.app/bin/mexext: Command not found

CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp

ld: library not found for -lboost_python

clang: error: linker command failed with exit code 1 (use -v to see invocation)

make: *** [python/caffe/_caffe.so] Error 1

在 2015年10月26日星期一 UTC-4上午10:39:27，Jan C Peters写道：

[Jan C Peters]

Hi,

well this error is due to another dependency of caffe, the boost library. You need to install that too. Did you follow the instructions on http://caffe.berkeleyvision.org/install_osx.html? I don't have any experience with development on a Mac, so I can't help you with that.

[Xi Yin]

Hi Jan, 

I did follow the instructions. I just tried to install dependency again. And it all showed to have been installed already. I tried the 'make pycaffe' again. It did not  give me any error. Here is the output.  Does it successfully compiled? I am not sure because usually there will be more things displaying while here I only get a few lines. Thanks. 

CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp

touch python/caffe/proto/__init__.py

PROTOC (python) src/caffe/proto/caffe.proto

Xis-MacBook-Pro:caffe-master xiyin$ make pycaffe

make: Nothing to be done for `pycaffe'.

在 2015年10月27日星期二 UTC-4上午4:10:42，Jan C Peters写道：

[Jan C Peters]

Yes, that is all it does: compiling the caffe proto for python and the _caffe backend python module, which is written in c++. You should be good to go!

P. S. Don't forget to add the caffe/python directory to your PYTHONPATH or something equivalent, or python will give you an ImportError when you do "import caffe".

[Xi Yin]

Great! Thank you so much for your help!  


在 2015年10月27日星期二 UTC-4上午10:22:37，Jan C Peters写道：

[Adam Erickson]

Thanks Jan! The ANACONDA_HOME fix still works for OS X and the latest Caffe.

Adam

[laven...@gmail.com]

Hi, would you help me?

I encountered the same problem when I did "make pycaffe".

The terminal shows that:

CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp

python/caffe/_caffe.cpp:1:10: fatal error: 'Python.h' file not found

#include <Python.h>  // NOLINT(bui

ld/include_alpha)

         ^

1 error generated.

And I have revised the Make.configure.

# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.

# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \

        -gencode arch=compute_20,code=sm_21 \

        -gencode arch=compute_30,code=sm_30 \

        -gencode arch=compute_35,code=sm_35 \

        -gencode arch=compute_50,code=sm_50 \

        -gencode arch=compute_52,code=sm_52 \

        -gencode arch=compute_60,code=sm_60 \

        -gencode arch=compute_61,code=sm_61 \

        -gencode arch=compute_61,code=compute_61

# BLAS choice:

# atlas for ATLAS (default)

# mkl for MKL

# open for OpenBlas

BLAS := atlas

# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.

# Leave commented to accept the defaults for your choice of BLAS

# (which should work)!

# BLAS_INCLUDE := /path/to/your/blas

# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path

BLAS_INCLUDE := /usr/local/opt/openblas/include

BLAS_LIB := /usr/local/opt/openblas/lib

# This is required only if you will compile the matlab interface.

# MATLAB directory should contain the mex binary in /bin.

# MATLAB_DIR := /usr/local

# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

# PYTHON_INCLUDE := /usr/include/python2.7 \

        /usr/lib/python2.7/dist-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

ANACONDA_HOME := /Users/lavenda/anaconda

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

$(ANACONDA_HOME)/include/python3.6 \

$(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include \

# Uncomment to use Python 3 (default is Python 2)

# PYTHON_LIBRARIES := boost_python3 python3.6m

# PYTHON_INCLUDE := /usr/include/python3.6m \

# /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.

#PYTHON_LIB := /usr/lib

PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)

# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include

# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)

# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.

LIBRARY_DIRS := /usr/local/opt/openblas/lib

iINCLUDE_DIRS := /usr/local/opt/openblas/include3.2/lib

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies

# INCLUDE_DIRS += $(shell brew --prefix)/include

# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)

# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)

# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.

# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)

# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`

BUILD_DIR := build

DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171

# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.

TEST_GPUID := 0

# enable pretty build (comment to see full commands)

Q ?= @

I'm nearly drove crazy by this. I almost tried everything.

Would you give me suggestions? Thanks!

[João Devezas]

Have you managed to solve this?

Thanks