how can i set the boost.python version to boost_python3?

[BK Zero]

in python 3.5, i make caffe successfully, but just cannot import caffe in python, as i said in:

https://groups.google.com/forum/#!topic/caffe-users/5_uAuyWOMgQ

and

https://groups.google.com/forum/#!topic/caffe-users/o4XbE0jC27E

thanks Jan's help, and i check the makefile and makefile.config today, and i see in makefile, the default PYTHON_LIBRARIES is set to PYTHON_LIBRARIES ?= boost_python python2.7.

so that's the problem in python3

if i comment "PYTHON_LIBRARIES := boost_python3 python3.5m" in makefile.config, when i import caffe, it will give a error message "ImportError: /home/zero/Documents/caffe-master/python/caffe/_caffe.so: undefined symbol: _ZN5boost6python6detail11init_moduleER11PyModuleDefPFvvE"

this means the python edition and the boost.python version mismatch.

so i should uncomment this line, but if i do this, when i make, i will got error msg like "can not find lboost_python3".

what should i do now? 

[Mladen Fernežir]

For me it's boost_python-py34 python3.4m

Possibly in your case, boost_python-py35 python3.5m will work.

[BK Zero]

thank you, but it gives me the error msg:

/usr/bin/ld: cannot find -lboost_python-py35

collect2: error: ld returned 1 exit status

make: *** [python/caffe/_caffe.so] Error 1

when i make pycaffe.

thanks anyway.

在 2016年4月28日星期四 UTC+8上午11:38:28，Mladen Fernežir写道：

[Mladen Fernezir]

Hmm, try locate boost_python* on your system. I had the same error as you which got solved with the correct boost_python name in Makefile.

Another matter is that you wrote that you are using Anaconda. Maybe there are some steps to account for that as well but I'm not sure.

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/_Iydbi8T-GQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/a0cb883c-f367-4d68-aa13-81c09a1297e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[BK Zero]

what command did you use? i try to find it by using

sudo find / -name boost_python*

but it tells me nothing at all...

在 2016年4月28日星期四 UTC+8下午2:26:47，Mladen Fernežir写道：

[BK Zero]

does that means i need some other packages, or maybe i never install boost correctly?

i installed boost by the following command:

sudo apt-get install libboost1.55-dev

is there anything wrong?

在 2016年4月28日星期四 UTC+8下午2:26:47，Mladen Fernežir写道：

Hmm, try locate boost_python* on your system. I had the same error as you which got solved with the correct boost_python name in Makefile.

[Mladen Fernežir]

I'm using locate boost_python and generally sudo updatedb to update the search database before that. In my case, the files are located in /usr/lib/x86_64-linux-gnu/. There's libboost_python-py27.so which links to libboost_python-py27.so.1.55.0 and libboost_python-py34.so which is a link to libboost_python-py34.so.1.55.0.

I used apt-get install libboost-all-dev to install boost, which is probably an overkill compared to just libboost1.55-dev.

[Jan]

libboost-all-dev just links to the default boost version for the current distribution, which seems to be 1.54 for Ubuntu 14.04. So it's not an overkill, it is just a meta package to draw in the default boost version.

Jan

[BK Zero]

i use libboost1.55-all-dev. and i remenber it has lboost_python and lboost_python3. but i haven't test this.

在 2016年4月28日星期四 UTC+8下午4:55:58，Jan写道：

[Mladen Fernežir]

I think that libboost-dev links to libboost1.54-dev, libbost-all-dev should be more than that.

http://packages.ubuntu.com/trusty/libboost-dev

http://packages.ubuntu.com/trusty/libdevel/libboost-all-dev

https://askubuntu.com/questions/308657/installing-boost-on-12-04-have-ran-sudo-apt-get-install-libboost-all-dev-now

[BK Zero]

thank you very much!

i see it, but mine is also libboost_python-py34. the error message gone. but when i run pytest, i got this

from google.protobuf.internal import enum_type_wrapper

ImportError: No module named 'google'

should i set some path like the PYTHONPATH? i have install libprotobuf, and i google it to find i should find the folder and run

sudo python setup.py install to link it to python.but i cannot find where i install it because i use the apt-get. can you give a more hint? thanks very much to solve my big problem with the boost :)

在 2016年4月28日星期四 UTC+8下午4:29:35，Mladen Fernežir写道：

[BK Zero]

thank you very much!

i see it, but mine is also libboost_python-py34. the error message gone. but when i run pytest, i got this

from google.protobuf.internal import enum_type_wrapper

ImportError: No module named 'google'

should i set some path like the PYTHONPATH? i have install libprotobuf, and i google it to find i should find the folder and run

sudo python setup.py install to link it to python.but i cannot find where i install it because i use the apt-get. can you give a more hint? thanks very much to solve my big problem with the boost :)

在 2016年4月28日星期四 UTC+8下午4:29:35，Mladen Fernežir写道：

I'm using locate boost_python and generally sudo updatedb to update the search database before that. In my case, the files are located in /usr/lib/x86_64-linux-gnu/. There's libboost_python-py27.so which links to libboost_python-py27.so.1.55.0 and libboost_python-py34.so which is a link to libboost_python-py34.so.1.55.0.

[Jan]

You need to install the python bindings of the protobuf library as well to really use the pycaffe interface (and run the pycaffe tests), install in Ubuntu by apt-get install python-protobuf. It has nothing to do with boost::python or PYTHONPATH.

Jan

[Mladen Fernežir]

Ah yes of course, it's a deep rabbit hole to install Caffe with Python 3. ;) Also, are you using Ubuntu 14.04 or have you moved to 15.10? You need version 3 protobuf for Python 3. There could be other issues as well. From previous notes:

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev python3-gflags

pip3 install protobuf

[Mladen Fernežir]

I forgot you were using Anaconda. Possibly this will work instead of pip3 install protobuf:

https://anaconda.org/pypi/protobuf-py3

pip install -i https://pypi.anaconda.org/pypi/simple protobuf-py3

This is 2.50 version and the link says it's deprecated so maybe you could just go with Anaconda's regular 2.6.1 package.

[BK Zero]

i use the ubuntu14.04LTS.

i have a question, if i apt-get install python-pip3, it will install to the default python3.4 or to my anaconda3.5?

and if i use a pip3 install protobuf, does it work for anaconda3?

o, i try it. and i face another problem. if i do "pip install protobuf" by the pip from anaconda, there are permission denied because i am not the root user. and if i do "sudo pip install protobuf" it seems like goes to another account and under that account pip links to python2.7 and it will return a msg "sudo: pip: command not found"

and i try the pip3, and the protobuf2.6 install to the python3.4

how can i fix it?

在 2016年4月28日星期四 UTC+8下午6:59:14，Mladen Fernežir写道：

[BK Zero]

and i try your links from anaconda.org, it says "HTTP error 404 while getting https://pypi.anaconda.org/pypi/simple/protobuf-py3/2.5.1/protobuf-py3-2.5.1.tar.gz (from https://pypi.anaconda.org/pypi/simple/protobuf-py3/)"

在 2016年4月28日星期四 UTC+8下午6:59:14，Mladen Fernežir写道：

I forgot you were using Anaconda. Possibly this will work instead of pip3 install protobuf:

[Jan]

Yes, in Ubuntu14.04 "pip" is the package manager for python 2.7, and "pip3" the one for python 3.x.

apt-get install python-pip3 has nothing to do with anaconda. It will just install pip3 into the regular python paths of your distribution (somewhere along /usr/lib/python3.4 or so).

The complete anaconda distribution of python is usually installed into a separate folder on your computer (in ~/anaconda maybe? Or if installed as root maybe in /usr/local/anaconda, at least these seem likely, I have no idea what the actual paths are). When you install a package through pip (or pip3) I think it depends which pip you are using, the one from Ubuntu or the anaconda one, where your package will be installed to. And the best is to really keep both python environments completely separate, but in practice confusions easily happen. This is why I prefer on Linux to use the built-in python distribution. Anaconda is nice for windows, but on any other OS it's simply not needed and will only lead to headaches. If you want/need to use version 3.x of python, that is still not a problem, nowadays python 2.7 and 3.x usually coexist peacefully in the repositories: python3 packages are usually prefixed with "python3-", while python 2.7 packages are only prefixed "python-". When you want to run the 3.x version of python simply run "pip3" or "python3" or "ipython3", instead of "pip" or "python" or "ipython".

Jan

[BK Zero]

apt-get install python-protobuf

this one doesn't work too...

should i give up using anaconda?

在 2016年4月28日星期四 UTC+8下午6:22:07，Jan写道：

[Jan]

I just noticed that "python3-protobuf" is not in the repositories. But you can install all these packages easily with pip. So in this case you can do

$ pip3 install protobuf

to get that package.

One more hint about pip: When I use pip (or pip3) I always add the --user flag, so pip installs the packages locally under my username, to not mess with the system-wide installed python packages installed through "apt-get install python-". They will still be found and used by python, but this way there is a clean separation between apt-get and pip. And you don't need sudo. This has proven a good strategy over the years.

Jan

[Jan]

If you don't have a good reason to use/keep using anaconda, get rid of it.

Jan

[BK Zero]

that's it!

--user is what i need! thus i can use the anaconda pip to install protobuf-py3. now i fix this, and face to a new one......when i make pytest, i got

Failed to include caffe_pb2, things might go wrong!

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/zero/Documents/caffe-master/python/caffe/__init__.py", line 4, in <module>

    from .proto.caffe_pb2 import TRAIN, TEST

  File "/home/zero/Documents/caffe-master/python/caffe/proto/caffe_pb2.py", line 1006, in <module>

    has_default_value=True, default_value=unicode("constant", "utf-8"),

NameError: name 'unicode' is not defined

maybe protobuf-py3 does not really support python3?

在 2016年4月28日星期四 UTC+8下午8:47:40，Jan写道：

[BK Zero]

ok, that's what i will do next. but i wonder which versioin of protobuf i should use?

in the caffe website, it says For protobuf support please install protobuf 3.0 alpha (https://developers.google.com/protocol-buffers/). 

so i will try one more time with anaconda3.

在 2016年4月28日星期四 UTC+8下午8:51:57，Jan写道：

[BK Zero]

so, if i do not use anaconda, which version of protobuf should i install? 

do you use pip3 install protobuf to the regular python3.4?

在 2016年4月28日星期四 UTC+8下午6:59:14，Mladen Fernežir写道：

I forgot you were using Anaconda. Possibly this will work instead of pip3 install protobuf:

[Jan]

Yes. Do pip3 install protobuf. And get rid of all the anaconda stuff, and comment out all the anaconda-related settings in Makefile.config, and recompile everything, especially pycaffe. Best also do a git pull before recompilation.

Jan

[BK Zero]

sorry to disturb you again.

i try to abandon anaconda and install caffe in the regular python3.4.

the make procedure seems sucessful, but when i make pytest, it gives me this message

ImportError: Failed to import test module: test_python_layer

Traceback (most recent call last):

  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests

    module = self._get_module_from_name(name)

  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name

    __import__(name)

  File "/home/zero/Documents/caffe-master/python/caffe/test/test_python_layer.py", line 6, in <module>

    import caffe

  File "caffe/__init__.py", line 1, in <module>

    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver

  File "caffe/pycaffe.py", line 11, in <module>

    import numpy as np

ImportError: No module named numpy

 it seems to run pytest from python2.7, but i am sure i configure the makefile.configure to python3.4. is it necessary to modify the python link from 2.7 to 3.4? does this help to fix the problem?

在 2016年4月28日星期四 UTC+8下午6:59:14，Mladen Fernežir写道：

I forgot you were using Anaconda. Possibly this will work instead of pip3 install protobuf:

[Mladen Fernezir]

Did you set WITH_PYTHON_LAYER in the  config file?

That being said, I didn't use make pytest. I did make all, make pycaffe, make test, make distribute and make runtest iirc.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/71c1ffe1-f40e-4f12-ab53-e5b83f8412f3%40googlegroups.com.

[BK Zero]

sorry, maybe pytest is not the issue. when i enter python3 and import caffe. receive an error msg from dateutil.rrule when import (rrule, MO, TU, WE, TH, FR, SA, SU, YEARLY,

i fix this by pip3 install dateuitl>=2.2, and then i face the problem:

from google.protobuf.internal import python_message

  File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 848

    except struct.error, e:

                       ^

SyntaxError: invalid syntax

i install the protobuf from the requirment.txt in /caffe/python, and use pip3 as

sudo pip3 install 'protobuf>=2.5.0'

i try to reinstall the protobuf automatically, but it doesn't work

try to install in version2.6, doesnot work too

and when i do sudo pip3 install 'protobuf>=3.0.0'

it cannot fetch it from web.

so which version of protobuf do you use?

在 2016年4月28日星期四 UTC+8下午6:59:14，Mladen Fernežir写道：

I forgot you were using Anaconda. Possibly this will work instead of pip3 install protobuf:

[BK Zero]

it stays comment. i do not quite understand what it means. you mean if i uncomment it, the pytest will not work?

but it doesn't matter because the import problem is not caused by it.

在 2016年4月28日星期四 UTC+8下午11:44:32，Mladen Fernežir写道：

[BK Zero]

someone ask the same question in github, i try to upgrade the six, but it doesn't work. and one gup successfully install caffe with anadonda3 2.5 with protobuf 3.0 beta2.

i am trying now. wish me to be lucky today.

and if there is something i do not know, please tell me.

thank you very much for helping me so much.

在 2016年4月28日星期四 UTC+8下午11:44:32，Mladen Fernežir写道：

Did you set WITH_PYTHON_LAYER in the  config file?

[Jan]

The WITH_PYTHON_LAYER causes the compilation of a mechanism that allows to define a layer in python, and use that in your network. It is much easier to handle than C++ and CUDA code, but also much less efficient. But for quick experiments . I am not 100% sure, but it may be that pytest also tries to test that and fails because you didn't compile it.

To the protobuf problem:

It seems there is a general problem with the protobuf library and Python 3.x, as there was/is (?) no support for Python 3.x in the protobuf library originally. I have no idea what the current state is. When looking at PyPI (pip's package repository), there seems to be no official protobuf package for Python 3. You can try one of the unofficial ones, such as https://pypi.python.org/pypi/python3-protobuf/2.5.0 .

Oh, I just found https://pypi.python.org/pypi/protobuf/3.0.0b2, which seems to be official and released for all python versions! Just make sure you also install the correct version (in this case 3.0) of the protobuf library (the pip installable packages are only the python bindings, not the library itself!). But then you may face the problem that protobuf 3.0 is not available in the ubuntu repos and/or not supported by caffe.

All in all it's just not worth the trouble. If you don't have a real need to use python 3, just stick with python 2.7, it has worked perfectly for me all the time.

Jan

[Mladen Fernezir]

Protobuf 3 does work and it is needed with Caffe for Python 3 support.

We use it at work, but yes, it's all a painful process to get it done.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/ea8d3b17-9603-4561-84cc-12708717d1df%40googlegroups.com.

[BK Zero]

thank you, i have make through this successfully few minutes ago.

it is the problem of the protobuf version. because i did apt-get install libprotobuf-dev earlier, when i try to install a new version for py3, it seems have some confiction of different version.

what confuse me is, i do uninstall libprotobuf-dev, but when i try

protoc --version

the old version of 2.5.0 is still there, so the complier cannot find my new protobuf.

在 2016年4月29日星期五 UTC+8下午4:02:06，Jan写道：

[BK Zero]

yes, you are right. i am too self-willed to insist on python3. it should be easier to use python2.7.

and sorry to bring a lot of trouble to you two, and a lot of thanks for your help.

在 2016年4月29日星期五 UTC+8下午4:02:06，Jan写道：

The WITH_PYTHON_LAYER causes the compilation of a mechanism that allows to define a layer in python, and use that in your network. It is much easier to handle than C++ and CUDA code, but also much less efficient. But for quick experiments . I am not 100% sure, but it may be that pytest also tries to test that and fails because you didn't compile it.

[BK Zero]

the painful process ended! thank you very much, i have learned a lot from you, and received a lot of help from you.

it is really happy to see no error message when i input "from caffe import layers" :)

在 2016年4月29日星期五 UTC+8下午4:22:59，Mladen Fernežir写道：

[Pitch]

 I am currently attempting to get Protobuf 3 working with Caffe. Where did you get the caffe.proto file for proto3? Did you edit the latest caffe.proto file from proto2 to proto3?