Re: [caffe-users] Compiling Caffe on Yosemite 10.10

[Evan Shelhamer]

Hey Kate,

This usually happens if the order of the includes and libraries are different. Check that the include and library paths in the Makefile.config and your LD_LIBRARY_PATH are in matching order. Note that Anaconda has its own version of HDF5 bundled in its libs + includes too.

The CPU_ONLY mode asking for nvcc is weird -- we'll look into that.

On Mon, Feb 9, 2015 at 17:12 Kate Rakelly <krak...@gmail.com> wrote:

This is a followup from Issue #1830 on Github (don't comment there! comment here! :)

I am on OS X Yosemite 10.10, using Anaconda Python, Boost 1.56, OpenCV 2.4.10 (using Homebrew to install everything) and caffe-dev.
I followed the installation instructions on the website for OS X 10.9, but used the new Makefile found in https://github.com/BVLC/caffe/commit/33a56e08d9a2714c2492bb7999a350636ed142be.

I am compiling CPU only version, and I do not have CUDA installed.

I am able to compile caffe, albeit with the warning

/bin/sh: /usr/local/cuda/bin/nvcc: No such file or directory

(standard_in) 1: parse error

because I do not have CUDA. (As I have uncommented the line CPU_ONLY := 1, it seems like it should not be looking for nvidia cuda at all.)

However, when I run make runtest, I get the error:

[ RUN      ] HDF5OutputLayerTest/1.TestForward

Warning! ***HDF5 library version mismatched error***

The HDF5 header files used to compile this application do not match

the version used by the HDF5 library to which this application is linked.

Data corruption or segmentation faults may occur if the application continues.

This can happen when an application was compiled by one version of HDF5 but

linked with a different version of static or shared HDF5 library.

You should recompile the application or check your shared library related

settings such as 'LD_LIBRARY_PATH'.

You can, at your own risk, disable this warning by setting the environment

variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'.

Setting it to 2 or higher will suppress the warning messages totally.

Headers are 1.8.13, library is 1.8.14

    SUMMARY OF THE HDF5 CONFIGURATION

    =================================

General Information:

-------------------

   HDF5 Version: 1.8.14

  Configured on: Thu Dec  4 22:14:51 GMT 2014

  Configured by: br...@miniyosemite.local

 Configure mode: production

    Host system: i386-apple-darwin14.0.0

      Uname information: Darwin miniyosemite.local 14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 x86_64

       Byte sex: little-endian

      Libraries: static, shared

     Installation point: /usr/local/Cellar/hdf5/1.8.14

Some more info...and then:

.build_release/test/test_all.testbin 0 --gtest_shuffle --gtest_filter="-*GPU*" && \

.build_release/test/test_all_dynamic_link.testbin 0 --gtest_shuffle --gtest_filter="-*GPU*"

dyld: Library not loaded: libhdf5_hl.8.dylib

  Referenced from: /Users/rakelly/caffe-dev/.build_release/test/test_all.testbin

  Reason: image not found

/bin/sh: line 1: 30205 Trace/BPT trap: 5       .build_release/test/test_all.testbin 0 --gtest_shuffle --gtest_filter="-*GPU*"

I have tried reinstalling hdf5, and then recompiling Caffe, with the same result.

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users+unsubscribe@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/2d928607-5f18-4810-b656-248afadabb77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kate Rakelly]

I have set my LD_LIBRARY_PATH to be identical to LIBRARY_DIRS in the Makefile.config

So it is LD_LIBRARY_PATH=/path/to/anaconda/lib:/usr/local/lib:/usr/lib

PYTHON_INCLUDE, PYTHON_LIB, INCLUDE_DIRS, and LIBRARY_DIRS in the Makefile.config are also all set with anaconda paths first.

I have verified that the library I want, libhdf5_hl.8.dylib, is located in /path/to/anaconda/lib.

And another version (the one that is probably causing the problem), libhdf5_hl.9.dylib is living in /usr/lib.

-Kate

[Kate Rakelly]

Update on this:

I managed to resolve the hdf5 library mismatch by forcing the linker to look in /path/to/anaconda/lib at runtime. This wasn't really a fix however as I then ran into a problem where it seemed to be trying to use two different pythons at once, resulting in this error: Fatal Python error: PyThreadState_Get: no current thread.

I decided to wipe everything and reinstall, this time without Anaconda to hopefully avoid the same linking errors.

I made sure to "brew install python" FIRST so that everything I installed with brew after that should link to brew python and not system python.

I followed the installation instructions, and set the $DYLD_FALLBACK_LIBRARY_PATH as suggested (note don't try setting any other DYLD paths as when you set one it no longer relies on defaults unless you set ALL of them).

Caffe compiles and all tests pass.

Pycaffe also compiles, but when I run make pytest (or try to import the caffe module) I get: 

cd python; python -m unittest discover -s caffe/test

/bin/sh: line 1: 28149 Segmentation fault: 11  python -m unittest discover -s caffe/test

make: *** [pytest] Error 139

I suspect this is actually a a failure to link masquerading as a seg fault but I am not sure how to debug this. 

Any help would be much appreciated.

Thanks.

-Kate

[Evan Shelhamer]

I've seen this kind of crash when boost::python and / or Caffe aren't compiled with the intended Python. It's hard to know exactly what's gone wrong, but you can check the linking by 

otool -L python/caffe/_caffe.so

python/caffe/_caffe.so:

python/caffe/_caffe.so (compatibility version 0.0.0, current version 0.0.0)
@rpath/libcaffe.so (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libglog.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/local/lib/libgflags.2.dylib (compatibility version 4.0.0, current version 4.0.0)
/usr/local/lib/libprotobuf.9.dylib (compatibility version 10.0.0, current version 10.1.0)
/usr/local/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libsnappy.1.dylib (compatibility version 4.0.0, current version 4.0.0)
/usr/local/lib/liblmdb.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
libhdf5_hl.8.dylib (compatibility version 9.0.0, current version 9.2.0)
libhdf5.8.dylib (compatibility version 9.0.0, current version 9.2.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/usr/local/lib/libopencv_core.2.4.dylib (compatibility version 2.4.0, current version 2.4.10)
/usr/local/lib/libopencv_highgui.2.4.dylib (compatibility version 2.4.0, current version 2.4.10)
/usr/local/lib/libopencv_imgproc.2.4.dylib (compatibility version 2.4.0, current version 2.4.10)
/usr/local/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/local/lib/libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
libpython2.7.dylib (compatibility version 2.7.0, current version 2.7.0)
libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

and check that everything lines up as intended (your output may differ from mine -- this is a 10.10 install with Anaconda Python).

Evan Shelhamer

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.

To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/CAL1-wrP3e4-sUnRzU-8PhBvZM%3DEa9HWN-Ks5Lf-AOu58%3DeLEZw%40mail.gmail.com.

[Kate Rakelly]

This verifies the problem, that it's using system python when everything brew installed (opencv, boost, etc.) is installed against brew python.

Maybe I should uninstall everything again, and install against system python?

python/caffe/_caffe.so:

python/caffe/_caffe.so (compatibility version 0.0.0, current version 0.0.0)

/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)

/usr/local/lib/libglog.0.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/local/lib/libgflags.2.dylib (compatibility version 4.0.0, current version 4.0.0)

/usr/local/opt/protobuf/lib/libprotobuf.9.dylib (compatibility version 10.0.0, current version 10.1.0)

/usr/local/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)

/usr/local/lib/libsnappy.1.dylib (compatibility version 4.0.0, current version 4.0.0)

/usr/local/lib/liblmdb.dylib (compatibility version 0.0.0, current version 0.0.0)

/usr/local/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)

/usr/local/lib/libhdf5_hl.9.dylib (compatibility version 10.0.0, current version 10.0.0)

/usr/local/lib/libhdf5.9.dylib (compatibility version 10.0.0, current version 10.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

/usr/local/lib/libopencv_core.2.4.dylib (compatibility version 2.4.0, current version 2.4.10)

/usr/local/lib/libopencv_highgui.2.4.dylib (compatibility version 2.4.0, current version 2.4.10)

/usr/local/lib/libopencv_imgproc.2.4.dylib (compatibility version 2.4.0, current version 2.4.10)

/usr/local/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)

/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0)

/usr/local/lib/libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)

/System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.6)

/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

-Kate

[Evan Shelhamer]

Maybe I should uninstall everything again, and install against system python?

No, that shouldn't be needed and could be yet more hassle. Are the include and library paths pointed to brew python in your Makefile.config? If all the other dependencies are against brew python you should be able to convince Caffe too.

By the way, whenever you switch dependencies or change Makefile.config settings it's best to `make clean` so that everything is on the same page. 

Evan Shelhamer

[Kate Rakelly]

Tried setting DYLD_FRAMEWORK_PATH to /usr/local/Cellar, no effect.

Ok I think you are right and the Makefile.config paths are the issue.

I am guessing and checking how they should be set.

This is how I think they should be:

I set the env variables like this:

PYTHON_INCLUDE := /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7:/usr/local/lib/python2.7/site-packages/numpy/core/include

(that first path is the exact path to the brew python headers)

INCLUDE_DIRS := /usr/local/include $(PYTHON_INCLUDE)

LIBRARY_DIRS := /usr/local/lib /usr/lib

This is flipped order from the default file, but I think it makes sense given we want brew python.

These settings won't compile unless I add /usr/include/python2.7 to INCLUDE_DIRS which then causes the whole problem since that is system python.

This is the default settings:

# PYTHON_INCLUDE := /usr/include/python2.7 \

                # /usr/local/lib/python2.7/site-packages/numpy/core/include

# PYTHON_LIB := /usr/lib

# INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include

# LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

(Side note, it is weird that in the default file, LIBRARY_DIRS is set to /usr/lib:/usr/local/lib:/usr/lib.)

-Kate

[Kate Rakelly]

My bad those settings do compile, I had a typo.

I have the same set fault error however.

-Kate

[Kate Rakelly]

Here is the traceback for the seg fault. 

It looks like the problem is in loading Boost

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread

0   ???                           000000000000000000 0 + 0

1   org.python.python             0x00000001028f80dd PyEval_GetGlobals + 23

2   org.python.python             0x000000010290762b PyImport_Import + 137

3   org.python.python             0x0000000102905d27 PyImport_ImportModule + 31

4   _caffe.so                     0x0000000101c43852 caffe::init_module__caffe() + 3330

5   libboost_python.dylib         0x0000000102817fd1 boost::python::handle_exception_impl(boost::function0<void>) + 81

6   libboost_python.dylib         0x0000000102819016 boost::python::detail::init_module(char const*, void (*)()) + 118

7   org.python.python             0x0000000100f434ea _PyImport_LoadDynamicModule + 140

8   org.python.python             0x0000000100f431d2 import_submodule + 270

9   org.python.python             0x0000000100f42d9c load_next + 280

10  org.python.python             0x0000000100f40f99 PyImport_ImportModuleLevel + 1135

11  org.python.python             0x0000000100f26459 builtin___import__ + 135

12  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

13  org.python.python             0x0000000100f30d9b PyEval_CallObjectWithKeywords + 93

14  org.python.python             0x0000000100f2d0b7 PyEval_EvalFrameEx + 8330

15  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

16  org.python.python             0x0000000100f2a86f PyEval_EvalCode + 54

17  org.python.python             0x0000000100f3fa46 PyImport_ExecCodeModuleEx + 241

18  org.python.python             0x0000000100f42731 load_source_module + 1091

19  org.python.python             0x0000000100f431d2 import_submodule + 270

20  org.python.python             0x0000000100f42d9c load_next + 280

21  org.python.python             0x0000000100f40f99 PyImport_ImportModuleLevel + 1135

22  org.python.python             0x0000000100f26459 builtin___import__ + 135

23  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

24  org.python.python             0x0000000100f30d9b PyEval_CallObjectWithKeywords + 93

25  org.python.python             0x0000000100f2d0b7 PyEval_EvalFrameEx + 8330

26  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

27  org.python.python             0x0000000100f2a86f PyEval_EvalCode + 54

28  org.python.python             0x0000000100f3fa46 PyImport_ExecCodeModuleEx + 241

29  org.python.python             0x0000000100f42731 load_source_module + 1091

30  org.python.python             0x0000000100f429f2 load_package + 303

31  org.python.python             0x0000000100f431d2 import_submodule + 270

32  org.python.python             0x0000000100f42d9c load_next + 280

33  org.python.python             0x0000000100f40f99 PyImport_ImportModuleLevel + 1135

34  org.python.python             0x0000000100f26459 builtin___import__ + 135

35  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

36  org.python.python             0x0000000100f30d9b PyEval_CallObjectWithKeywords + 93

37  org.python.python             0x0000000100f2d0b7 PyEval_EvalFrameEx + 8330

38  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

39  org.python.python             0x0000000100f2a86f PyEval_EvalCode + 54

40  org.python.python             0x0000000100f3fa46 PyImport_ExecCodeModuleEx + 241

41  org.python.python             0x0000000100f42731 load_source_module + 1091

42  org.python.python             0x0000000100f431d2 import_submodule + 270

43  org.python.python             0x0000000100f42d9c load_next + 280

44  org.python.python             0x0000000100f40f99 PyImport_ImportModuleLevel + 1135

45  org.python.python             0x0000000100f26459 builtin___import__ + 135

46  org.python.python             0x0000000100f2e9af PyEval_EvalFrameEx + 14722

47  org.python.python             0x0000000100f3163e fast_function + 262

48  org.python.python             0x0000000100f2e3ca PyEval_EvalFrameEx + 13213

49  org.python.python             0x0000000100ec68a5 gen_send_ex + 193

50  org.python.python             0x0000000100ed259a listextend + 297

51  org.python.python             0x0000000100ed37c4 list_init + 97

52  org.python.python             0x0000000100ef3bcf type_call + 182

53  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

54  org.python.python             0x0000000100f2de8c PyEval_EvalFrameEx + 11871

55  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

56  org.python.python             0x0000000100f315ad fast_function + 117

57  org.python.python             0x0000000100f2e3ca PyEval_EvalFrameEx + 13213

58  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

59  org.python.python             0x0000000100f315ad fast_function + 117

60  org.python.python             0x0000000100f2e3ca PyEval_EvalFrameEx + 13213

61  org.python.python             0x0000000100f3163e fast_function + 262

62  org.python.python             0x0000000100f2e3ca PyEval_EvalFrameEx + 13213

63  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

64  org.python.python             0x0000000100ecf31c function_call + 350

65  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

66  org.python.python             0x0000000100ebc1a5 instancemethod_call + 174

67  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

68  org.python.python             0x0000000100ef82c9 slot_tp_init + 64

69  org.python.python             0x0000000100ef3bcf type_call + 182

70  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

71  org.python.python             0x0000000100f2de8c PyEval_EvalFrameEx + 11871

72  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

73  org.python.python             0x0000000100f2f1c6 PyEval_EvalFrameEx + 16793

74  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

75  org.python.python             0x0000000100f315ad fast_function + 117

76  org.python.python             0x0000000100f2e3ca PyEval_EvalFrameEx + 13213

77  org.python.python             0x0000000100f2adf9 PyEval_EvalCodeEx + 1412

78  org.python.python             0x0000000100ecf31c function_call + 350

79  org.python.python             0x0000000100eb13b6 PyObject_Call + 99

80  org.python.python             0x0000000100f5c019 RunModule + 113

81  org.python.python             0x0000000100f5bb01 Py_Main + 2429

82  libdyld.dylib                 0x00007fff870365c9 start + 1

"otool -L libboost_python.dylib" reveals

libboost_python.dylib:

/usr/local/lib/libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)

/usr/local/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)

/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

-Kate

[Anand Sampat]

Can you elaborate on where exactly the the version of HDF5 version is specified in the includes and libraries? I am trying the same thing (Caffe on Yosemite) and have the same error, but it seems it would be easiest to resolve this mismatch by just changing a requirement rather than reinstalling using a different python (my understanding is Anaconda is the preferred choice anyways). 

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.

[郑海永]

The CPU_ONLY error happens because of the following code in Makefile:

 CUDA_VERSION := $(shell $(CUDA_DIR)/bin/nvcc -V | grep -o 'release \d' | grep -o '\d')

I think you'd better change this to:

233 ifeq ($(OSX), 1)

234         CXX := /usr/bin/clang++

235         ifneq ($(CPU_ONLY), 1)

236                 CUDA_VERSION := $(shell $(CUDA_DIR)/bin/nvcc -V | grep -o 'release \d' | grep -o '\d')

237                 ifeq ($(shell echo $(CUDA_VERSION) \< 7.0 | bc), 1)

238                         CXXFLAGS += -stdlib=libstdc++

239                         LINKFLAGS += -stdlib=libstdc++

240                 endif

241         endif

在 2015年2月10日星期二 UTC+8下午2:59:43，Evan Shelhamer写道：

To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.

[Evan Shelhamer]

Fixed in https://github.com/BVLC/caffe/pull/2224 thanks to @small-yellow-duck

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/6c442190-2e2f-4054-86d8-4688bcd19067%40googlegroups.com.