Pycaffe not working: No module named google.protobuf.internal

[Ramon Pires]

I have tried several times to install pycaffe. It seems to be all right. However, something is missing. Always gives the same error with the protobuf.

>>> import caffe

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/user/bin/caffe/python/caffe/__init__.py", line 1, in <module>

    from .pycaffe import Net, SGDSolver

  File "/home/user/bin/caffe/python/caffe/pycaffe.py", line 14, in <module>

    import caffe.io

  File "/home/user/bin/caffe/python/caffe/io.py", line 8, in <module>

    from caffe.proto import caffe_pb2

  File "/home/user/bin/caffe/python/caffe/proto/caffe_pb2.py", line 4, in <module>

    from google.protobuf.internal import enum_type_wrapper

ImportError: No module named google.protobuf.internal

Note that the PYTHONPATH is properly configured. And I have the protobuf (see below):

Note that the PYTHONPATH is properly configured. And I have the protobuf (see below):

/usr/include/google/protobuf

/usr/include/google/protobuf/descriptor.h

/usr/include/google/protobuf/descriptor.pb.h

/usr/include/google/protobuf/descriptor.proto

...

/usr/include/google/protobuf/stubs/platform_macros.h

/usr/include/google/protobuf/stubs/template_util.h

/usr/include/google/protobuf/stubs/type_traits.h

In Makefile.config, I included the "/usr/include" dir:

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include

But something is wrong. Someone can help me, please?

Bests,

[CaffeStudent]

have you installed the google protobuf package?  Usually your package manager will have a version of "protobuf" that you can install.

[Ramon Pires]

Yep. Not by me (because I have no permission), but the admin ensured me that all the dependences were properly installed following the script http://caffe.berkeleyvision.org/install_apt.html.

In addition, I can see that the protobuf is on /usr/include/google/protobuf.

[CaffeStudent]

My PYTHON_INCLUDE in Makefile.config, looks like this:

PYTHON_INCLUDE := /usr/include/python2.7 \

                /usr/lib/python2.7/dist-packages/numpy/core/include \

                /usr/lib64/python2.7/site-packages/numpy/core/include

Could it be that your admin installed the required pacakges under a python3 distribution rather than the python2.7 distribution?  Or maybe your include path points to a symbolic link which is pointing to a python distribution different from your admin install?

If you just run python at the command prompt, and you type "import caffe", or "import google.protobuf.internal", do those load properly?  What version of python does it say you are using?

[Ramon Pires]

Thanks for the answer, CaffeStudent.

The PYTHON_INCLUDE is similar to yours:

PYTHON_INCLUDE := /usr/include/python2.7 \

                /usr/lib/python2.7/dist-packages/numpy/core/include

And in the command prompt, the python version that have been used in 2.7.6., as I already expected.

Importing "caffe" or "google.protobuf.internal", the error message is the same.

Do you have an ideia of what is missing?

[CaffeStudent]

you could try doing a "pip install protobuf", it seems like something isn't quite right with your protobuf installation.  You might also try rebuilding "pycaffe".

[Ramon Pires]

I have no permission to install. But I can try to contact the admin.

Could you check, please, in your machine where is the google/protobuf? Is it in /usr/local/lib/python2.7/dist-packages/?

I tried to install in my local machine and copy to /home/user/lib/python2.7/dist-packages/ in the cluster. I adjusted the PYTHON_INCLUDE in Makefile.config, but no success.

[CaffeStudent]

/usr/lib/python2.7/site-packages/google/protobuf
/usr/include/google/protobuf

[Ramon Pires]

Thanks, man.

I sent a message to the admin asking for reinstall the protobuf, and am awaiting a response.

[Ramon Pires]

Resolved. The admin finally considered to reinstall the google.protobuf, as I asked. This was enough.

[Platon George]

Installing 'pip install protobuf' did the trick for me :-)

[Jason Borne]

Well, recently I also met the question which was really confusing to me. I was sure I had installed protobuf successfully, but there was always importerror of google.protobuf.internal. After many reinstalls, I doubted the cause of error may be due to Anaconda. So I used "/home/username/anaconda2/bin/pip", instead of merely "pip", to install protobuf again. Luckily, it worked!

I hope my experience can help you if your cause of error happens to be similar with mine.

[Jan C Peters]

@Jason, just for the record:

This is because you have two separate python ecosystem installations on your machine: the one provided by your linux distribution (which is used when you just call "pip") and the one installed and maintained by anaconda (which is used when you call "/home/username/anaconda2/bin/pip"). Apparently you got the import error when working with anaconda's python (and/or using anaconda's python env for building pycaffe). Having protobuf installed in your linux distribution's standard paths obviously will not change anything in this case.

By the way: Anaconda is great to install a well-provisioned python env on windows, but for linux it doesn't really make much sense imho. Every popular linux distribution already provides a working python env which can be easily extended by your distro's package manager or pip.

Jan

[Dora]

Hi Caffe Student,

I'm also encountering same problem. "import caffe" says:

# Python3 will most likely not be able to load protobuf
# Generated by the protocol buffer compiler.  DO NOT EDIT!
ImportError: No module named google.protobuf


Can you please help me with this??

[Tiêu Phong Võ Đình]

You can install it locally: pip install protobuf --user

[Hello]

I am able to import no problem when i run "sudo python" followed by "import pycaffe" but i have this problem when i do NOT use "sudo". Why is this? 

[Jason Lee]

Please check whether you have two python in your system. e.g. /usr/bin and /usr/local/bin

在 2016年4月18日星期一 UTC-7下午10:55:58，Hello写道：

[Xiangxue Wang]

Though the problem might be solved for most of people, I still want to post more detail about it since my situation is little bit different.

For sure, my problem is indeed caused from two/several python environments. I have several python available by linux module packages. As suggested by caffe installation site, I choose anaconda for better environment management.

First of all, import caffe in the python terminal gave the same error as above post. The error indicate the caffe python wrapper is exactly in the right PYTHONPATH, so is google.protobuf. But not the google.protobuf.internal. 

[xxw345@gpu002t caffe]$ python

Python 2.7.12 |Anaconda custom (64-bit)| (default, Jul  2 2016, 17:42:40)

[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

Anaconda is brought to you by Continuum Analytics.

Please check out: http://continuum.io/thanks and https://anaconda.org

>>> import google.protobuf

>>> import caffe

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/xxw345/caffe/python/caffe/__init__.py", line 1, in <module>

    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver

  File "/home/xxw345/caffe/python/caffe/pycaffe.py", line 15, in <module>

    import caffe.io

  File "/home/xxw345/caffe/python/caffe/io.py", line 8, in <module>

    from caffe.proto import caffe_pb2

  File "/home/xxw345/caffe/python/caffe/proto/caffe_pb2.py", line 6, in <module>

    from google.protobuf.internal import enum_type_wrapper

ImportError: No module named internal

Then, I go step further to print where the python founds the google.protobuf, shown as below:

[xxw345@gpu002t caffe]$ python

Python 2.7.12 |Anaconda custom (64-bit)| (default, Jul  2 2016, 17:42:40)

[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

Anaconda is brought to you by Continuum Analytics.

Please check out: http://continuum.io/thanks and https://anaconda.org

>>> import google.protobuf

>>> print google.protobuf.__file__

/home/xxw345/.local/lib/python2.7/site-packages/google/protobuf/__init__.pyc

We can see it's in ~/.local/lib/python2.7/.......... This might indicate where the error came from. Since in this case, I used anaconda, it's supposed that all python module should be reasonably installed under the anaconda directory. The reason that python found the protobuf in my personal folder is probably due to I previously installed myself in that folder. But at this time, I only want to use anaconda version. So I did pip install protobuf to see if that's helped:

[xxw345@gpu002t caffe]$ which protoc

~/.usr/local/bin/protoc

[xxw345@gpu002t caffe]$ ~/anaconda2/bin/pip install protobuf

Requirement already satisfied (use --upgrade to upgrade): protobuf in /home/xxw345/.local/lib/python2.7/site-packages

Requirement already satisfied (use --upgrade to upgrade): six>=1.9 in /home/xxw345/anaconda2/lib/python2.7/site-packages (from protobuf)

Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/xxw345/anaconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg (from protobuf)

However, it did nothing if only ran the pip install commnad, it will output that protobuf already in my home personal foler.

So I first uninstall protobuf to purge the protobuf environment. (If this not help, you can even delete protobuf and google/protobuf in the personal python site-package folder

) 

[xxw345@gpu002t caffe]$ ~/anaconda2/bin/pip uninstall protobuf

Uninstalling protobuf-3.0.0:

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0-py2.7-nspkg.pth

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0.dist-info/DESCRIPTION.rst

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0.dist-info/METADATA

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0.dist-info/RECORD

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0.dist-info/WHEEL

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0.dist-info/metadata.json

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0.dist-info/namespace_packages.txt

  /home/xxw345/.local/lib/python2.7/site-packages/protobuf-3.0.0.dist-info/top_level.txt

Proceed (y/n)? y

  Successfully uninstalled protobuf-3.0.0

Then reinstall protobuf:

[xxw345@gpu002t caffe]$ ~/anaconda2/bin/pip install protobuf

Collecting protobuf

  Using cached protobuf-3.0.0-py2.py3-none-any.whl

Requirement already satisfied (use --upgrade to upgrade): six>=1.9 in /home/xxw345/anaconda2/lib/python2.7/site-packages (from protobuf)

Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/xxw345/anaconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg (from protobuf)

Installing collected packages: protobuf

Successfully installed protobuf-3.0.0

Try again to see where the protobuf be installed:

[xxw345@gpu002t caffe]$ ~/anaconda2/bin/pip install protobuf

Requirement already satisfied (use --upgrade to upgrade): protobuf in /home/xxw345/anaconda2/lib/python2.7/site-packages

Requirement already satisfied (use --upgrade to upgrade): six>=1.9 in /home/xxw345/anaconda2/lib/python2.7/site-packages (from protobuf)

Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/xxw345/anaconda2/lib/python2.7/site-packages/setuptools-23.0.0-py2.7.egg (from protobuf)

The protobuf now is successfully installed in anaconda folder, let's open python and try to import caffe again! Done!

[xxw345@gpu002t caffe]$ python

Python 2.7.12 |Anaconda custom (64-bit)| (default, Jul  2 2016, 17:42:40)

[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

Anaconda is brought to you by Continuum Analytics.

Please check out: http://continuum.io/thanks and https://anaconda.org

>>> import caffe

>>>

[Tong Chen]

You really solved my problem. but is it possible to use both Anaconda and my Linux distribution so that i don't need to reinstall many things in anaconda like protobuf. 

Thanks so much!  

在 2016年1月14日星期四 UTC+8下午8:58:37，Jan写道：

[Neil Kronlage]

I had the same problem on Ubuntu and was able to fix it by running:

    sudo apt-get install python-protobuf