OSX 10.12 make error

[Andy Tsang]

Hi guys. I am using OSX 10.12, with two versions of python 2.7 (brew and anaconda).

I have been trying serval tutorials already, which I also found lots of them are outdated. And only this brings me farthest up to this stage.

However, I have encounter the following error

AR -o .build_release/lib/libcaffe.a

LD -o .build_release/lib/libcaffe.so.1.0.0-rc3

clang: warning: argument unused during compilation: '-pthread'

ld: library not found for -lboost_python

clang: error: linker command failed with exit code 1 (use -v to see invocation)

make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1

make: *** Waiting for unfinished jobs....

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_conv_layer.o) has no symbols

 

Getting some clues from other post, I have already tried to check my env-PATH, PYTHONPATH ; makefile.config - PYTHON_INCLUDE,BLAS_INCLUDE,BLAS_LIB,PYTHON_LIB

I should have covered most of the variables is linked to valid path, still nothing changes.

I had run "make clean" after every change before re-run the make command. 

I had only found this post seems directly similar to my situation, but there isn't any solution. Kind of stuck in the installation part for a few days already....

Here is my config

## Refer to http://caffe.berkeleyvision.org/installation.html

# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).

# USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).

# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers

# USE_OPENCV := 0

USE_LEVELDB := 0

# USE_LMDB := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)

# You should not set this flag if you will be reading LMDBs with any

# possibility of simultaneous read and write

# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3

# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.

# N.B. the default for Linux is g++ and the default for OSX is clang++

# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.

CUDA_DIR := /usr/local/cuda

# On Ubuntu 14.04, if cuda tools are installed via

# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:

# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.

# For CUDA < 6.0, comment the *_50 lines for compatibility.

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \

-gencode arch=compute_20,code=sm_21 \

-gencode arch=compute_30,code=sm_30 \

-gencode arch=compute_35,code=sm_35 \

-gencode arch=compute_50,code=sm_50 \

-gencode arch=compute_50,code=compute_50

# BLAS choice:

# atlas for ATLAS (default)

# mkl for MKL

# open for OpenBlas

BLAS := atlas

# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.

# Leave commented to accept the defaults for your choice of BLAS

# (which should work)!

BLAS_INCLUDE := /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/Headers

BLAS_LIB := /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A 

# Homebrew puts openblas in a directory that is not on the standard search path

# BLAS_INCLUDE := $(shell brew --prefix openblas)/include

# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.

# MATLAB directory should contain the mex binary in /bin.

# MATLAB_DIR := /usr/local

# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.

# We need to be able to find Python.h and numpy/arrayobject.h.

PYTHON_INCLUDE := /usr/include/python2.7 \

     /Library/Python/2.7/site-packages/numpy/core/include

# Anaconda Python distribution is quite popular. Include path:

# Verify anaconda location, sometimes it's in root.

ANACONDA_HOME := $(HOME)/anaconda

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

$(ANACONDA_HOME)/include/python2.7 \

$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# Uncomment to use Python 3 (default is Python 2)

# PYTHON_LIBRARIES := boost_python3 python3.5m

# PYTHON_INCLUDE := /usr/include/python3.5m \

#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.

PYTHON_LIB := /usr/lib

# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)

# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include

# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)

WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies

# INCLUDE_DIRS += $(shell brew --prefix)/include

# LIBRARY_DIRS += $(shell brew --prefix)/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.

# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)

# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`

BUILD_DIR := build

DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171

# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.

TEST_GPUID := 0

# enable pretty build (comment to see full commands)

Q ?= @

[Przemek D]

I know little about compilation under OSX but the error log says you're missing a boost_python library. Also, looks like you're using an outdated version of caffe, 1.0.0-rc3, while the current standard is 1.0.0 - consider updating.

[Andy Tsang]

well u mention boost_python I remember I had came cross it in one of the tutorial previously.

However, I was stuck on that as well, I use the formula provided by it, and I also replace sha1 to sha256. But i got either sha256 mismatch or forgot some other error. I tried some other solution still not working.

Another thing is I cant restore the formula as well, keep getting some sort of "keg" error? Would be nice if anyone has a clu on this problem as well.

One more thing, I remember is I tried to install the dependency "build-essential" as well, but it doesn't work as well, from what I knew its from linux and brew don't have it. So I install apt-get via fink, still cannot found that dependency.(As so post mention I have tried to update, still not working)

About the caffe version, I thought it was already the latest? I clone from "https://github.com/BVLC/caffe.git"

[Andy Tsang]

Just an update, by recovering brew boost and boost_python formula. I was able to install both of the successfully.

However, the about the outdated version of caffe, I still not figure it out yet. Suppose the version I clone from git should be the latest?

[Przemek D]

Yes the git clone should give you the latest version. Sorry about the confusion, the error you received (make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1) contains the old version number. If you run caffe --version in terminal you should see what version you have (newest is 1.0.0).

[Andy Tsang]

Well, I have tried sth about the protobuf. I have brew installed 3.3.0 and using this to install 2.6.1 as one of the tutorial said.

If i point it to 3.3.0 it returns other error even I tried to change the caffe.proto

cd ~/Documents  
curl -LO https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.bz2  
tar xvjf protobuf-2.6.1.tar.bz2  
rm protobuf-2.6.1.tar.bz2  
cd protobuf-2.6.1  
./configure
make  
make check  
sudo make install  

But now I get some other "stuff" with "make all"

AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0


clang: warning: argument unused during compilation: '-pthread'


/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_conv_layer.o) has no symbols


/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_lcn_layer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_lrn_layer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_pooling_layer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_relu_layer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_sigmoid_layer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_softmax_layer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_tanh_layer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(parallel.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(db_leveldb.o) has no symbols
CXX/LD -o .build_release/tools/caffe.bin
CXX/LD -o .build_release/tools/compute_image_mean.bin
CXX/LD -o .build_release/tools/convert_imageset.bin
CXX/LD -o .build_release/tools/device_query.bin
CXX/LD -o .build_release/tools/extract_features.bin
CXX/LD -o .build_release/tools/finetune_net.bin
CXX/LD -o .build_release/tools/net_speed_benchmark.bin
CXX/LD -o .build_release/tools/test_net.bin


clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'


CXX/LD -o .build_release/tools/train_net.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_text.bin
CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin


clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'


CXX/LD -o .build_release/examples/cifar10/convert_cifar_data.bin
CXX/LD -o .build_release/examples/cpp_classification/classification.bin
CXX/LD -o .build_release/examples/mnist/convert_mnist_data.bin
CXX/LD -o .build_release/examples/siamese/convert_mnist_siamese_data.bin


clang: warning: argument unused during compilation: '-pthread'


clang: clang: warning: warningargument unused during compilation: '-pthread':


argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'

 

Not sure what causing it, and when I run "make pytest"

I have experienced two situation 

a) What I get previously - Python quite unexpectedly. ( sth like that

b) What I get now

    I have checked the .so file, it existed correctly under the path it mention not found.

ERROR: test_solver (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_solver
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/Users/ATC/Documents/caffe/python/caffe/test/test_solver.py", line 7, in <module>
    import caffe
  File "caffe/__init__.py", line 1, in <module>
    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver, NCCL, Timer
  File "caffe/pycaffe.py", line 13, in <module>
    from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: dlopen(caffe/_caffe.so, 2): Library not loaded: @rpath/libhdf5_hl.10.dylib
  Referenced from: /Users/ATC/Documents/caffe/python/caffe/_caffe.so
  Reason: image not found
----------------------------------------------------------------------
Ran 10 tests in 0.000s

FAILED (errors=10)
make: *** [pytest] Error 1

Kind of not sure whats more can be done. 

[Andy Tsang]

Another update, problem still not solved.

I tried not to use anaconda at all. I did pass test && runtest. But I faced a problem of "Python quit unexpectedly" when I try to run pytest. 

I found some post mentioning using different version of python for make and test, however, I have one version available only. I tried with PYTHON_LAYER OPTION as this post, nothing changes.

For more details

I wish someone can help to verify my config on PYTHON_INCLUDE and any missing thing in $PATH

PYTHON_INCLUDE := /usr/include/python2.7 \

     /usr/local/lib/python2.7/site-packages/numpy/core/include

 

$PATH = /Developer/NVIDIA/CUDA-8.0.61/bin:/sw/bin:/sw/sbin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/ATC/anaconda/bin:/usr/local/share/python:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/opt/local/bin 

Meanwhile, here is my "make all" result, I saw people suggest to ignore warnings but I am not totally sure about is it correctly build?

AR -o .build_release/lib/libcaffe.a

LD -o .build_release/lib/libcaffe.so.1.0.0

clang: warning: argument unused during compilation: '-pthread'

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_conv_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_lcn_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_lrn_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_pooling_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_relu_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_sigmoid_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_softmax_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_tanh_layer.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(parallel.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn.o) has no symbols

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(db_leveldb.o) has no symbols

CXX/LD -o .build_release/tools/caffe.bin

CXX/LD -o .build_release/tools/compute_image_mean.bin

CXX/LD -o .build_release/tools/convert_imageset.bin

CXX/LD -o .build_release/tools/device_query.bin

CXX/LD -o .build_release/tools/extract_features.bin

CXX/LD -o .build_release/tools/finetune_net.bin

CXX/LD -o .build_release/tools/net_speed_benchmark.bin

CXX/LD -o .build_release/tools/test_net.bin

clangclang: : warningwarning: : argument unused during compilation: '-pthread'argument unused during compilation: '-pthread'clang: 

warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

CXX/LD -o .build_release/tools/train_net.bin

CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin

CXX/LD -o .build_release/tools/upgrade_net_proto_text.bin

CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin

clang: clang: warning: warningargument unused during compilation: '-pthread': argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

CXX/LD -o .build_release/examples/cifar10/convert_cifar_data.bin

CXX/LD -o .build_release/examples/cpp_classification/classification.bin

CXX/LD -o .build_release/examples/mnist/convert_mnist_data.bin

CXX/LD -o .build_release/examples/siamese/convert_mnist_siamese_data.bin

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

clang: warning: argument unused during compilation: '-pthread'

On Monday, 8 May 2017 14:37:41 UTC+8, Andy Tsang wrote:

[Nathan Ing]

A few things: 1) make sure your Makefile.config points at the correct cuda / cudnn.h location. And that these are the most updated versions from NVIDIA. I assume this is all in order since runtest passes but just in case.. 

2) Have you tried a virtualenv to build caffe in? I found that it solves lots of headaches with the python interface on OS X, especially when you have multiple versions of python. You'll want to specify which python to build to virtualenv with, and provide the correct paths in PYTHON_INCLUDE and PYTHON_LIBS. I have not used Anaconda python to make Caffe, only Homebrew python + numpy.

It's a headache but well worth learning about the inner workings of the compiler and makefiles if you're planning to build these development packages much in the future. Especially how to decipher the errors and know what they're missing. You can ignore the -pthread warnings, but the "no symbols" i would be concerned about. Good luck!

[Przemek D]

This looks like an issue that was already discussed in the following thread. It is a bit dated, but maybe worth a try?

[Andy Tsang]

Thx for the reply.

I have tried virtualenv with the following command, I have tried couple setting with the env variables. But all of them return the same result, which is error139.

After summarising serval tutorial, I found that each of them teach to point to python all over the places

Still my TensorFlow, Ipython, opencv was function normally when I use anaconda

But I suppose it might be the cause of some problems? 

Meanwhile, my python --verions is 2.7.9 but I think some of them are not, e.g. from brew /Cellar python is 2.7.13 (I just installed today for virtualenv, I was not using python in Cellar before, but still they return same error.)

I am not sure how to fix it... which should I keep and which does the PYTHON_INCLUDE and PYTHON_LIBS should normally point to????

Directory having python:

/usr/local/Cellar/python/2.7.13
/usr/include/python2.7
/usr/lib/python2.7
/usr/bin/python2.7
/Library/Python/2.7/site-packages
/usr/local/lib/python2.7/site-packages/ 
/System/Library/Frameworks/Python.framework/Versions/2.7

 

And I don't know which should point to ......./site-packages/numpy/core/include/ 

vitrualenv command:

virtualenv venv --system-site-packages 

source venv/bin/activate 

virtualenv -p /usr/local/bin/python2.7 /Users/ATC/Documents/caffe

 

Env last used:

 PATH = /Users/ATC/Documents/caffe/venv/bin:/Developer/NVIDIA/CUDA-8.0.61/bin:/sw/bin:/sw/sbin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/ATC/anaconda/bin:/Users/ATC/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/opt/local/bin

 PYTHONPATH = /Library/Python/2.7/site-packages:/usr/local/lib/python2.7/site-packages/ 

export PYTHON_INCLUDE=/usr/local/lib/python2.7/site-packages/numpy/core/include/:/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/include/python2.7
export PYTHON_LIBS=/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib

Full Error:

cd python; python -m unittest discover -s caffe/test
/bin/sh: line 1: 34667 Segmentation fault: 11  python -m unittest discover -s caffe/test
make: *** [pytest] Error 139

About the no symbol problem, I have tried to read the post @Przemek D suggested, I also found some other post but all of them were either told to post back in this user-group or not solved.

Referring to the post Przemek suggested, they mention its sth about OpenCV link issue, some people suggested to change USE_libstdcpp:BOOL=FALSE  in CMakeCache.txt. 

But same as some others I don't have the CMakeCache.txt in my build directory, one person did mention toggle the advanced settings using ccmake to find but I not sure how to do that??