Issue 275 in protobuf: Incorrect installation with setup.py breaks local override
prot...@googlecode.com
Owner: ken...@google.com
Labels: Type-Defect Priority-Medium
New issue 275 by zbe...@google.com: Incorrect installation with setup.py
breaks local override
http://code.google.com/p/protobuf/issues/detail?id=275
What steps will reproduce the problem?
1. Unpack the distribution somewhere
2. cd protobuf-2.3.0/python
3. ./setup.py install --root=./x
4. File ./x/usr/lib64/python2.6/site-packages/google/__init__.py (path
varies) is missing, but python/google/__init__.py (in the sources) exists.
A little background:
Some chromium tests use local version of protobuf, and override sys.path to
include the local version instead of the system package. However, as soon
as the system package is installed, the override stops working, and system
protobuf is always pulled in.
There seems to be certain amount of black magic involved, but we tracked
down the issue to the above __init__.py file missing.
This affects (at the very least) gentoo, whose ebuilds are used to install
protobuf as a system package in ChromiumOS, but the bug comes from protobuf
setup.
prot...@googlecode.com
Cc: the.real...@gmail.com
Comment #2 on issue 275 by liuj...@google.com: Incorrect installation with
setup.py breaks local override
http://code.google.com/p/protobuf/issues/detail?id=275
This is strange. When the --root is not given in the setup.py, the library
will be packed into an egg, which contains the __init__.py in each
directory level. After setting the --root, it won't pack an egg, and the
__init__.py in the root is missing.
P.J, would you have any ideas about this?
prot...@googlecode.com
Comment #3 on issue 275 by the.real...@gmail.com: Incorrect installation
http://code.google.com/p/protobuf/issues/detail?id=275
This is intended and correct behavior for installs using --root and
namespace packages.
A --root install is not something that's supposed to be run where you
install it; --root installs are for making system packages like rpms or
msi's or what-have-you, that will then be actually installed to a final
target location.
In the case of a namespace package, this means that __init__.py for
namespace packages can't be included in the install, as it would create
installation conflicts when you install more than one RPM (or whatever) for
the same namespace package. (See PEP 382 for more background.)
Setuptools works around this by using .pth files to replace the
__init__.py, but the tradeoff is that the directory where the package is
installed *must* support loading of .pth files by Python.
When you install a project without using --root, the install process checks
that the target directory does in fact support .pth files, but this check
isn't done with --root because by definition, a --root install is not going
to be used as-is -- it's going to be chrooted or packed up in some sort of
archive and then installed.
(That's why you're not getting any warnings -- as far as setuptools is
concerned, everything's fine, because /usr/lib64/python2.6/site-packages/
*is* a .pth-supporting directory, and it's expected that when you actually
use the install, you'll be running it from there, *without* the /x in front
of it.)
So, what it boils down to is that either you need to stop using --root, or
else you need to configure your Python to process .pth files in the
/x/usr/lib64/python2.6/site-packages directory (where /x is your --root).
You can do this with:
site.addsitedir('/x/usr/lib64/python2.6/site-packages')
Which will also add the directory to sys.path. (In other words, you can
just change your test code to add the path this way.)
Another option is that if you're using --root in order to avoid the use of
an .egg directory or file, you could perhaps use the 'pip' tool instead,
maybe with a virtualenv. virtualenv and pip are well-established tools for
creating isolated environments with local overrides to systemwide packages,
without using .egg files or directories.
Either way, however, I'd personally just avoid using --root in the first
place; it's really not intended that you'd ever directly use a distutils
--root install for *anything* except building an installation archive,
chroot jail, or symlink farm. That's what it's always been for in
distutils, and setuptools doesn't change that.
In summary, your options are (from least change to most):
* Use site.addsitedir() instead of just tacking the dummy site-packages on
sys.path
* Stop using --root and just install .egg(s) to a temporary directory on
PYTHONPATH
* Use pip and virtualenv (without --root)
By the way, please note that I don't normally respond to direct emails for
support requests on setuptools; in future please use the normal mailing
list (distutils-sig) and bug tracker ( http://bugs.python.org/setuptools/ )
contact channels, so that the information is available for others to see
and search as well. Thanks!
prot...@googlecode.com
Status: OutOfScope
Comment #4 on issue 275 by liuj...@google.com: Incorrect installation with
setup.py breaks local override
http://code.google.com/p/protobuf/issues/detail?id=275
Thanks P.J!
Zdenek, looks like this isn't a protobuf specific problem. Would you please
follow P.J's advice and change your local chrome tests? (e.g. do not use
--root). Thanks.
prot...@googlecode.com
Comment #5 on issue 275 by zbe...@google.com: Incorrect installation with
http://code.google.com/p/protobuf/issues/detail?id=275
Thanks for the explanation.
I believe this needs a little clarification though:
The local chrome tests don't use protobuf installed with --root. They just
have the live in a third_party/protobuf/ directory and they are installed
using an embedded .gyp file.
(http://src.chromium.org/svn/trunk/src/third_party/protobuf/protobuf.gyp,
look for "'target_name': 'py_proto'," to see the self-explanatory build
rules.)
This installation seems "correct", as it has the __init__.py both in the
namespace (google) and the package (protobuf). It does not, however, use
setuptools or distutils, it just copies files, so there may be some issue
here. But:
The global system package (installed by the linux distribution) is
installed using setuptools with --root, which is correct according to what
P.J said, because it is, afterall, a distribution package (not necessarily
an rpm). It also contains the protobuf-$version-$pyversion-nspkg.pth file.
And in that enviroment, overriding the system package seems impossible. The
code overrides PYTHONPATH prior to running the .py files, and import
google.protobuf always pulls the wrong one, it is always the system package
that gets imported, no matter what. This is a summary that one of the
developers digging in this wrote about it in email discussion:
----snip----
To cut a long story short:
/usr/local/lib/python2.6/site-packages/protobuf-2.3.0-py2.6-nspkg.pth
exists and is processed by the 'site' module (which is automatically
imported by Python). That pth contains this wonderful code:
import sys,types,os; p =
os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',)); ie =
os.path.exists(os.path.join(p,'__init__.py')); m = not ie and
sys.modules.setdefault('google',types.ModuleType('google')); mp = (m
or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and
mp.append(p)
Which basically tests for
/usr/local/lib/python2.6/site-packages/google/__init__.py (which
doesn't exist) and, because it doesn't exist, adds a built in 'google'
package to the modules list. Once the modules list already contains
'google', sys.path is ignored when trying to import submodules (like
the protobuf stuff) because the module is "already loaded".
I have no idea what that pth file is trying to achieve, but I touched
/usr/local/lib/python2.6/site-packages/google/__init__.py and now
things appear to be working, on that host at least.
----snip----
From the PEP:
"The import statement is extended so that it directly considers *.pth files
during import; a directory is considered a package if it either contains a
file named __init__.py, or a file whose name ends with ".pth". Unlike .pth
files on the top level, lines starting with "import" are not supported in
per-package .pth files."
This makes sense, but both of the places that we want to import from are
valid in this sense (one has a .pth, the other has an __init__.py), and one
of them always takes precedence (the one with .pth).
So, I'm not sure if this is an OutOfScope, but I'm certainly still confused
about what we (or someone else) did wrong.
prot...@googlecode.com
Comment #6 on issue 275 by the.real...@gmail.com: Incorrect installation
http://code.google.com/p/protobuf/issues/detail?id=275
Ah. Based on this information, the problem is that your local install is
*not* using setuptools for the install. As a result, the protobuf package
doesn't end up being added to the google.__path__. (From the earlier
message, I thought you were using --root to install the "local" copy rather
than the "system" copy.)
There are two options to fix this:
1. Use setuptools to install the local version of protobuf
2. Use "import google;
google.__path__.insert(0,'/the/local/protobuf/google')", in place of
inserting the local copy on sys.path.
While making the system-level __init__.py is a workaround for this
particular use case, you should be aware that it may introduce *different*
importing problems for other libraries under the google.* package
namespace, depending on what is imported when, and what order things are
added on to sys.path. So I don't recommend that you keep that workaround.
prot...@googlecode.com
Comment #7 on issue 275 by zbe...@google.com: Incorrect installation with
http://code.google.com/p/protobuf/issues/detail?id=275
Thanks. This does not make me too happy, because it's quite a bit of magic
associated with something as simple as overriding a system package locally,
but your detailed explanation is very appreciated.
prot...@googlecode.com
Comment #8 on issue 275 by the.real...@gmail.com: Incorrect installation
http://code.google.com/p/protobuf/issues/detail?id=275
Well, setuptools handles the magic for you if you use it for the local
installation. Conversely, if you use virtualenv, it'll similarly handle
the magic for you. (I think it does, anyway.)
Pretty much, the process of managing namespace packages *is* magic right
now, so you just have to choose which magician (setuptools,
virtualenv, ...) is going to handle it for you, or else handle all the path
munging yourself. (Which is even more unpleasant... and is why the tools
exist.)
That's what PEP 382 is about, btw -- getting rid of the hackery that's
currently needed for namespace packages to work, and replacing it with a
builtin declarative mechanism, to replace all the old imperative "magic"
tools.
Unfortunately, PEP 382 isn't implemented yet. Maybe you can get somebody
at Google to work on it as a 20% project? ;-)
prot...@googlecode.com
Comment #9 on issue 275 by ravishan...@gmail.com: Incorrect installation
Traceback (most recent call last):
File "C:\Python27\Scripts\mercury", line 44, in <module>
__import__("mwr.mercury.%s" % (sys.argv[1]))
File "C:\Python27\lib\site-packages\mwr\mercury\console.py", line 7, in
<module>
from mwr.droidhg.console import Console
File "C:\Python27\lib\site-packages\mwr\droidhg\__init__.py", line 1, in
<module>
from mwr.droidhg.device import Device, Devices
File "C:\Python27\lib\site-packages\mwr\droidhg\device.py", line 1, in
<module>
from mwr.cinnibar.api.builders import SystemRequestFactory
File "C:\Python27\lib\site-packages\mwr\cinnibar\api\__init__.py", line
9, in <module>
from frame import Frame
File "C:\Python27\lib\site-packages\mwr\cinnibar\api\frame.py", line 3,
in <module>
from mwr.cinnibar.api.protobuf_pb2 import Message
File "C:\Python27\lib\site-packages\mwr\cinnibar\api\protobuf_pb2.py",
line 6, in <module>
from google.protobuf import descriptor_pb2
ImportError: cannot import name descriptor_pb2
would you have any ideas about this?
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings