[Distutils] Odd limitation in packaging/distutils2
Paul Moore
PS D:\Data\PyPI> D:\Data\cpython\PCbuild\python -m packaging.run install nose
installing third-party projects from an uninstalled Python is not supported
And yet after downloading the nose source...
PS D:\Data\PyPI> bsdtar xzf D:\Downloads\Python\nose-1.1.2.tar.gz
PS D:\Data\PyPI> cd .\nose-1.1.2
PS D:\Data\PyPI\nose-1.1.2> D:\Data\cpython\PCbuild\python -m
packaging.run install
Installing from source directory: 'D:\\Data\\PyPI\\nose-1.1.2'
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz
Extracting in c:\users\gustav\appdata\local\temp\tmpecso_w
Now working in c:\users\gustav\appdata\local\temp\tmpecso_w\distribute-0.6.14
Building a Distribute egg in D:\Data\PyPI\nose-1.1.2
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
creating build
creating build\src
... etc
(The build actually failed, but for reasons unrelated to the fact that
it's an uninstalled Python, as far as I can see).
Why not allow installing direct from PyPI?
Paul.
_______________________________________________
Distutils-SIG maillist - Distut...@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Éric Araujo
> PS D:\Data\PyPI> D:\Data\cpython\PCbuild\python -m packaging.run install nose
> installing third-party projects from an uninstalled Python is not supported
There is a comment in the source to explain that:
# Python would try to install into the site-packages directory under
# $PREFIX, but when running from an uninstalled code checkout we don't
# want to create directories under the installation root
(Original bug report: #12246)
> And yet after downloading the nose source...
> [snip]
Ah, I need to reopen the bug. The patch edited d2.install but forgot to
touch the install_commands.
Regards
Paul Moore
> Hi Paul,
>
>> PS D:\Data\PyPI> D:\Data\cpython\PCbuild\python -m packaging.run install nose
>> installing third-party projects from an uninstalled Python is not supported
> There is a comment in the source to explain that:
>
> # Python would try to install into the site-packages directory under
> # $PREFIX, but when running from an uninstalled code checkout we don't
> # want to create directories under the installation root
>
> (Original bug report: #12246)
>
>> And yet after downloading the nose source...
>> [snip]
> Ah, I need to reopen the bug. The patch edited d2.install but forgot to
> touch the install_commands.
I'm still a bit confused. This is on Windows, and maybe things are
different there, but installing seems to just work for me (except for
the case where things are to be downloaded). What problem should I
look for?
The reason I care is that installing packages into a dev build is
really useful for testing packaging changes. I'm not actually sure how
to turn a dev build into an installed build on Windows...
Paul.
Éric Araujo
> On 2 November 2011 14:18, Éric Araujo <mer...@netwok.org> wrote:
> I'm still a bit confused. This is on Windows, and maybe things are
> different there, but installing seems to just work for me (except for
> the case where things are to be downloaded). What problem should I
> look for?
On UNIX, the sysconfig module exposes information from the configure
script, such as the install prefix. In an uninstalled built checkout,
this install prefix defaults to /usr/local, so trying to install a
distribution would create /usr/local/lib/python3.3/site-packages, which
would be incorrect and polluting. That’s the problem I wanted to prevent.
Can you define what you mean with “just work”? Does it install things
into $checkout/Lib/site-packages?
> The reason I care is that installing packages into a dev build is
> really useful for testing packaging changes. I'm not actually sure how
> to turn a dev build into an installed build on Windows...
http://docs.python.org/devguide/#quick-start : “On Windows, load the
project file PCbuild\pcbuild.sln in Visual Studio, select Debug, and
Build -> Build Solution;”
Cheers
Paul Moore
> Can you define what you mean with “just work”? Does it install things
> into $checkout/Lib/site-packages?
Seems to. Certainly does for my own packages.
>> The reason I care is that installing packages into a dev build is
>> really useful for testing packaging changes. I'm not actually sure how
>> to turn a dev build into an installed build on Windows...
>
> http://docs.python.org/devguide/#quick-start : “On Windows, load the
> project file PCbuild\pcbuild.sln in Visual Studio, select Debug, and
> Build -> Build Solution;”
Sorry, I wasn't clear. That I know. But that produces what I assume
counts as a "non-installed" build (with python.exe in the PCBuild
directory).
Looking at is_python_build, it's just checking for the existence of
Modules\Setup.*, so I suppose I could just rename or delete those
files. It all seems a bit arbitrary, though.
On a non-installed Python build (i.e., python built in a checkout) on
Windows, purelib and site-packages look fine:
PS D:\Data> D:\Data\cpython\PCbuild\python.exe
Python 3.3.0a0 (default, Oct 28 2011, 19:56:45) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig, os
>>> sysconfig.get_path('purelib')
'D:\\Data\\cpython\\Lib\\site-packages'
>>> os.path.isdir(sysconfig.get_path('purelib'))
True
>>>
It looks like this might be a Unix-only issue - if so, can I suggest
that the restriction be removed on Windows, as it *is* useful to be
able to install packages in a non-installed Python build.
I'd argue that on Unix, get_path('purelib') should find the right
site-packages even for an uninstalled build. But I've no idea if
that's too hard, or is somehow otherwise an issue. And I don't need
the functionality myself, so I'll leave that one for the Unix users to
decide.
Paul
Éric Araujo
>> Can you define what you mean with “just work”? Does it install things
>> into $checkout/Lib/site-packages?
> Seems to. Certainly does for my own packages.
Oh, nice. The site-packages directory is not in .hgignore, so I think
this works by a happy accident.
>>> The reason I care is that installing packages into a dev build is
>>> really useful for testing packaging changes.
Agreed! With virtual environments coming to the standard library, I
think the ability to install projects from an uninstalled Python will
become even more useful. I will reopen the bug and try to fix sysconfig
on UNIX (#12141, #6087) to remove the limitation for all OSes.
>>> I'm not actually sure how to turn a dev build into an installed build
>>> on Windows...
>> http://docs.python.org/devguide/#quick-start : “On Windows, load the
>> project file PCbuild\pcbuild.sln in Visual Studio, select Debug, and
>> Build -> Build Solution;”
>
> Sorry, I wasn't clear. That I know. But that produces what I assume
> counts as a "non-installed" build (with python.exe in the PCBuild
> directory).
It’s me, I confused building and installing. *puts stupid hat on*
I don’t know if Visual Studio can install a project, or if you have to
turn it into a installer somehow first and then click it. You could ask
on IRC or python-dev.
> Looking at is_python_build, it's just checking for the existence of
> Modules\Setup.*, so I suppose I could just rename or delete those
> files. It all seems a bit arbitrary, though.
I wouldn’t do that: It would trick Python into thinking it’s installed
without actually doing all the things that make it installed.
Regards
Paul Moore
> I wouldn’t do that: It would trick Python into thinking it’s installed
> without actually doing all the things that make it installed.
Agreed, it's not good. It's just that I don't know (on Windows) what
things *do* make a Python build "installed". I'll see if I can find a
definitive statement from somewhere...
Paul.
Mark Hammond
> On 3 November 2011 16:45, Éric Araujo<mer...@netwok.org> wrote:
>> I wouldn’t do that: It would trick Python into thinking it’s installed
>> without actually doing all the things that make it installed.
>
> Agreed, it's not good. It's just that I don't know (on Windows) what
> things *do* make a Python build "installed". I'll see if I can find a
> definitive statement from somewhere...
There really isn't anything that makes it "installed". A built Python
works completely fine just where it is (and indeed, will work just fine
if moved somewhere else). The closest thing to being "installed" on
windows is an "InstalledPath" (or something :) registry key for the
version, but that is only necessary in a limited number of contexts -
when some executable other than the in-place python[w].exe needs to know
where things are.
I think the original report is certainly a bug on Windows (and seems
mis-guided on other platforms, but each to their own :)
Mark
Paul Moore
> There really isn't anything that makes it "installed". A built Python works
> completely fine just where it is (and indeed, will work just fine if moved
> somewhere else). The closest thing to being "installed" on windows is an
> "InstalledPath" (or something :) registry key for the version, but that is
> only necessary in a limited number of contexts - when some executable other
> than the in-place python[w].exe needs to know where things are.
So that would imply that sysconfig.is_python_build is basically
meaningless on Windows. So at a minimum, it should always return False
on Windows.
Actually, it's not part of the public API of sysconfig (it's not in
__all__) and it's only used in one place within sysconfig.py, so maybe
it should be removed and the check done inline in that one place...
Paul.
Éric Araujo
Le 04/11/2011 13:11, Mark Hammond a écrit :
> On 4/11/2011 3:53 AM, Paul Moore wrote:
>> Agreed, it's not good. It's just that I don't know (on Windows) what
>> things *do* make a Python build "installed". I'll see if I can find a
>> definitive statement from somewhere...
>
> There really isn't anything that makes it "installed". A built Python
> works completely fine just where it is (and indeed, will work just fine
> if moved somewhere else). The closest thing to being "installed" on
> windows is an "InstalledPath" (or something :) registry key for the
> version, but that is only necessary in a limited number of contexts -
> when some executable other than the in-place python[w].exe needs to know
> where things are.
Ah, okay. The situation is different on UNIX: Installing Python
typically writes the file in read-only mode (one consequence: users have
to call setup.py --user, use virtualenv or switch to the superuser to
install things), and does not just copy the build dir (so missing data
files in the Makefile cause bugs that can only be seen when running the
test suite or other code from an installed Python).
> I think the original report is certainly a bug on Windows (and seems
> mis-guided on other platforms, but each to their own :)
I’ve already said that I want to remove the limitation for all OSes :)
Cheers
Ned Deily
Éric Araujo <mer...@netwok.org> wrote:
> Le 04/11/2011 13:11, Mark Hammond a écrit :
> > On 4/11/2011 3:53 AM, Paul Moore wrote:
> >> Agreed, it's not good. It's just that I don't know (on Windows) what
> >> things *do* make a Python build "installed". I'll see if I can find a
> >> definitive statement from somewhere...
> >
> > There really isn't anything that makes it "installed". A built Python
> > works completely fine just where it is (and indeed, will work just fine
> > if moved somewhere else). The closest thing to being "installed" on
> > windows is an "InstalledPath" (or something :) registry key for the
> > version, but that is only necessary in a limited number of contexts -
> > when some executable other than the in-place python[w].exe needs to know
> > where things are.
>
> Ah, okay. The situation is different on UNIX: Installing Python
> typically writes the file in read-only mode (one consequence: users have
> to call setup.py --user, use virtualenv or switch to the superuser to
> install things), and does not just copy the build dir (so missing data
> files in the Makefile cause bugs that can only be seen when running the
> test suite or other code from an installed Python).
One other very important difference: if the build was configured as
--enable-shared, the final installed path to the shared library is
usually captured in the built python executable. On most Unixy
platforms that means you *cannot* reliably run the python executable
from the build directory without taking some steps to override the
default dynamic library search path, i.e. usually LD_LIBRARY_PATH. In
the best case, the interpreter fails to start; in the worst, you get
deceptive behavior when the newly built interpreter executable actually
runs with a previously installed shared library. This is also true for
OS X framework builds, which are a special kind of shared-library builds.
--
Ned Deily,
n...@acm.org