[Distutils] distribute's sandboxing doesn't preserve working_set; leads to setup_requires problems
Erik Bray
I've got a tricky problem I'm trying to deal with. Here's the scenario:
I'm trying to build a package that has two requirements in
setup_requires; let's say `setup_requires = ['package_A',
'package_B']`.
The problem is that "package_B" also contains "package_A" in its own
setup_requires.
What happens when I do an install/build is that package_B is
downloaded and installed--in the process it also downloads and
installs package_A in its sandbox. This wouldn't be a problem except
that the sandboxed temp install of package_A is left in
pkg_resources.working_set.entries, and so it's assumed package_A is
already installed, and the main install doesn't try to fetch it.
However, when it goes to import from package_A, I get an import error
(because the sandbox it was installed in has since been deleted).
If I reverse the order it doesn't work either for a different, but
related problem. package_A gets installed into the cwd and is added
to pkg_resources.working_set. When package_B is installed it uses the
existing package_A installation, so that's fine. But then, due to
some complicated interplay, package_A never gets added to sys.path.
I'd consider this a defect, though I'm wondering if there's something
I could do differently to avoid this situation.
Thanks,
Erik
_______________________________________________
Distutils-SIG maillist - Distut...@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Erik Bray
> Hello all,
> I've got a tricky problem I'm trying to deal with. Here's the scenario:
>
> I'm trying to build a package that has two requirements in
> setup_requires; let's say `setup_requires = ['package_A',
> 'package_B']`.
> The problem is that "package_B" also contains "package_A" in its own
> setup_requires.
>
> What happens when I do an install/build is that package_B is
> downloaded and installed--in the process it also downloads and
> installs package_A in its sandbox. This wouldn't be a problem except
> that the sandboxed temp install of package_A is left in
> pkg_resources.working_set.entries, and so it's assumed package_A is
> already installed, and the main install doesn't try to fetch it.
> However, when it goes to import from package_A, I get an import error
> (because the sandbox it was installed in has since been deleted).
>
> If I reverse the order it doesn't work either for a different, but
> related problem. package_A gets installed into the cwd and is added
> to pkg_resources.working_set. When package_B is installed it uses the
> existing package_A installation, so that's fine. But then, due to
> some complicated interplay, package_A never gets added to sys.path.
>
> I'd consider this a defect, though I'm wondering if there's something
> I could do differently to avoid this situation.
Just to confirm my theory about this, I modified
setuptools.sandbox.run_setup to also save off and restore
pkg_resources.working_set.{entries,entry_keys,by_key}. This solves
the problem, and the build works regardless of how my setup_requires
are ordered. Is there any reason this shouldn't be done? It seems
reasonable to me...
It would also still be nice to find a workaround until and unless this
gets patched. All I can think of at the moment is monkey-patching :/
P.J. Eby
>Hello all,
>I've got a tricky problem I'm trying to deal with. Here's the scenario:
>
>I'm trying to build a package that has two requirements in
>setup_requires; let's say `setup_requires = ['package_A',
>'package_B']`.
>The problem is that "package_B" also contains "package_A" in its own
>setup_requires.
>
>What happens when I do an install/build is that package_B is
>downloaded and installed--in the process it also downloads and
>installs package_A in its sandbox. This wouldn't be a problem except
>that the sandboxed temp install of package_A is left in
>pkg_resources.working_set.entries, and so it's assumed package_A is
>already installed, and the main install doesn't try to fetch it.
>However, when it goes to import from package_A, I get an import error
>(because the sandbox it was installed in has since been deleted).
>
>If I reverse the order it doesn't work either for a different, but
>related problem. package_A gets installed into the cwd and is added
>to pkg_resources.working_set. When package_B is installed it uses the
>existing package_A installation, so that's fine. But then, due to
>some complicated interplay, package_A never gets added to sys.path.
>
>I'd consider this a defect, though I'm wondering if there's something
>I could do differently to avoid this situation.
One such workaround would be to use setuptools, since I fixed this
bug a couple of years ago.
(Apparently, the fix still hasn't made it to Distribute.)
P.J. Eby
>Just to confirm my theory about this, I modified
>setuptools.sandbox.run_setup to also save off and restore
>pkg_resources.working_set.{entries,entry_keys,by_key}. This solves
>the problem, and the build works regardless of how my setup_requires
>are ordered. Is there any reason this shouldn't be done? It seems
>reasonable to me... It would also still be nice to find a workaround
>until and unless this gets patched.
That's necessary, but I don't think it's sufficient. The changes I
did to fix this in 2009 save other aspects of pkg_resources state
besides just those, and I don't think it's something you can fix by
monkeypatching.
(I could be wrong, of course, but given that setuptools was already
been fixed, I'm not that motivated to investigate more deeply.)
Tarek Ziadé
I have added an issue about this here:
https://bitbucket.org/tarek/distribute/issue/205/distributes-sandboxing-doesnt-preserve
Since we're plannning a release this week-end for the upcoming python
3 release, I'll see if I can add a patch for this problem
Thanks
>
> Thanks,
> Erik
> _______________________________________________
> Distutils-SIG maillist - Distut...@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
--
Tarek Ziadé | http://ziade.org
chiggsy
P.J. Eby
>Are people still using setuptools?
Yes. Over the last 9 hours alone, the 0.6 development snapshot
version (0.6c12dev_r88795) was downloaded from over 100 unique IPs.
They most likely represent people manually upgrading to the latest
version, since the user agents are mostly older setuptools versions,
like 0.6c5, 0.6c9 and 0.6c11.
> I thought that distribute was the new way forward...?
Not really. Unless you're using Python 3, or you want different
default options from setuptools, there's little advantage to using
distribute. (It also includes bugs that setuptools does not.)
In addition, the announced direction of distribute is that they're
replacing it with the new "packaging" project, formerly known as distutils2.
Which means, again, that outside of Python 3, there's no compelling
reason to switch; you might as well wait for "packaging" to roll around.
So, all the hype was pretty much just that: hype and FUD-slinging.
Sebastien Douche
> So, all the hype was pretty much just that: hype and FUD-slinging.
Stop! We are tired of reading your perpetual whining.
--
Sebastien Douche <sdo...@gmail.com>
Twitter: @sdouche (agile, lean, python, git, open source)
David Stanek
Not really. Unless you're using Python 3, or you want different default options from setuptools, there's little advantage to using distribute. (It also includes bugs that setuptools does not.)
--
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek