[Python-Dev] Python environment registration in the Windows Registry
339 views
Skip to first unread message
Steve Dower
Feb 3, 2016, 12:17:11 AM2/3/16
to pytho...@python.org
I was throwing around some ideas with colleagues about how we detect
Python installations on Windows from within Visual Studio, and it came
up that there are many Python distros that install into different
locations but write the same registry entries. (I knew about this, of
course, but this time I decided to do something.)
Apart from not being detected properly by all IDEs/tools/installers,
non-standard distros that register themselves in the official keys may
also mess with the default sys.path values. For example, at one point
(possibly still true) if you installed both Canopy and Anaconda, you
would break the first one because they tried to load the other's stdlib.
Other implementations have different structures or do not register
themselves at all, which also makes it more complicated for tools to
discover them.
So here is a rough proposal to standardise the registry keys that can be
set on Windows in a way that (a) lets other installers besides the
official ones have equal footing, (b) provides consistent search and
resolution semantics for tools, and (c) includes slightly more rich
metadata (such as display names and URLs). Presented in PEP-like form
here, but if feedback suggests just putting it in the docs I'm okay with
that too. It is fully backwards compatible with official releases of
Python (at least back to 2.5, possibly further) and does not require
modifications to Python or the official installer - it is purely
codifying a superset of what we already do.
Any and all feedback welcomed, especially from the owners of other
distros, Python implementations or tools on the list.
Cheers,
Steve
-----
PEP: ???
Title: Python environment registration in the Windows Registry
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower <steve...@python.org>
Status: Draft
Type: ???
Content-Type: text/x-rst
Created: 02-Feb-2016
Abstract
========
When installed on Windows, the official Python installer creates a
registry key for discovery and detection by other applications.
Unofficial installers, such as those used by distributions, typically
create identical keys for the same purpose. However, these may conflict
with the official installer or other distributions.
This PEP defines a schema for the Python registry key to allow
unofficial installers to separately register their installation, and to
allow applications to detect and correctly display all Python
environments on a user's machine. No implementation changes to Python
are proposed with this PEP.
The schema matches the registry values that have been used by the
official installer since at least Python 2.5, and the resolution
behaviour matches the behaviour of the official Python releases.
Specification
=============
We consider there to be a single collection of Python environments on a
machine, where the collection may be different for each user of the
machine. There are three potential registry locations where the
collection may be stored based on the installation options of each
environment. These are::
HKEY_CURRENT_USER\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag>
On a given machine, an environment is uniquely identified by its
Company-Tag pair. Keys should be searched in the order shown, and if the
same Company-Tag pair appears in more than one of the above locations,
only the first occurrence is offerred.
Official Python releases use ``PythonCore`` for Company, and the value
of ``sys.winver`` for Tag. Other registered environments may use any
values for Company and Tag. Recommendations are made in the following
sections.
Backwards Compatibility
-----------------------
Python 3.4 and earlier did not distinguish between 32-bit and 64-bit
builds in ``sys.winver``. As a result, it is possible to have valid
side-by-side installations of both 32-bit and 64-bit interpreters.
To ensure backwards compatibility, applications should treat
environments listed under the following two registry keys as distinct,
even if Tag matches::
HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>
Note that this does not apply to Python 3.5 and later, which uses
different Tags. Environments registered under other Company names must
use distinct Tags for side-by-side installations.
1. Environments in ``HKEY_CURRENT_USER`` are always preferred
2. Environments in ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` are
preferred if the interpreter is known to be 32-bit
Company
-------
The Company part of the key is intended to group related environments
and to ensure that Tags are namespaced appropriately. The key name
should be alphanumeric without spaces and likely to be unique. For
example, a trademarked name, a UUID, or a hostname would be appropriate::
HKEY_CURRENT_USER\Software\Python\ExampleCorp
HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60
HKEY_CURRENT_USER\Software\Python\www.example.com
If a string value named ``DisplayName`` exists, it should be used to
identify the environment category to users. Otherwise, the name of the
key should be used.
If a string value named ``SupportUrl`` exists, it may be displayed or
otherwise used to direct users to a web site related to the environment.
A complete example may look like::
HKEY_CURRENT_USER\Software\Python\ExampleCorp
(Default) = (value not set)
DisplayName = "Example Corp"
SupportUrl = "http://www.example.com"
Tag
---
The Tag part of the key is intended to uniquely identify an environment
within those provided by a single company. The key name should be
alphanumeric without spaces and stable across installations. For
example, the Python language version, a UUID or a partial/complete hash
would be appropriate; an integer counter that increases for each new
environment may not::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66
If a string value named ``DisplayName`` exists, it should be used to
identify the environment to users. Otherwise, the name of the key should
be used.
If a string value named ``SupportUrl`` exists, it may be displayed or
otherwise used to direct users to a web site related to the environment.
If a string value named ``Version`` exists, it should be used to
identify the version of the environment. This is independent from the
version of Python implemented by the environment.
If a string value named ``SysVersion`` exists, it must be in ``x.y`` or
``x.y.z`` format matching the version returned by ``sys.version_info``
in the interpreter. Otherwise, if the Tag matches this format it is
used. If not, the Python version is unknown.
Note that each of these values is recommended, but optional. A complete
example may look like this::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66
(Default) = (value not set)
DisplayName = "Distro 3"
SupportUrl = "http://www.example.com/distro-3"
Version = "3.0.12345.0"
SysVersion = "3.6.0"
InstallPath
-----------
Beneath the environment key, an ``InstallPath`` key must be created.
This key is always named ``InstallPath``, and the default value must
match ``sys.prefix``::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath
(Default) = "C:\ExampleCorpPy36"
If a string value named ``ExecutablePath`` exists, it must be a path to
the ``python.exe`` (or equivalent) executable. Otherwise, the
interpreter executable is assumed to be called ``python.exe`` and exist
in the directory referenced by the default value.
If a string value named ``WindowedExecutablePath`` exists, it must be a
path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the
windowed interpreter executable is assumed to be called ``pythonw.exe``
and exist in the directory referenced by the default value.
A complete example may look like::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\InstallPath
(Default) = "C:\ExampleDistro30"
ExecutablePath = "C:\ExampleDistro30\ex_python.exe"
WindowedExecutablePath = "C:\ExampleDistro30\ex_pythonw.exe"
Help
----
Beneath the environment key, a ``Help`` key may be created. This key is
always named ``Help`` if present and has no default value.
Each subkey of ``Help`` specifies a documentation file, tool, or URL
associated with the environment. The subkey may have any name, and the
default value is a string appropriate for passing to ``os.startfile`` or
equivalent.
If a string value named ``DisplayName`` exists, it should be used to
identify the help file to users. Otherwise, the key name should be used.
A complete example may look like::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\Help
Python\
(Default) = "C:\ExampleDistro30\python36.chm"
DisplayName = "Python Documentation"
Extras\
(Default) = "http://www.example.com/tutorial"
DisplayName = "Example Distro Online Tutorial"
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com
Python installations on Windows from within Visual Studio, and it came
up that there are many Python distros that install into different
locations but write the same registry entries. (I knew about this, of
course, but this time I decided to do something.)
Apart from not being detected properly by all IDEs/tools/installers,
non-standard distros that register themselves in the official keys may
also mess with the default sys.path values. For example, at one point
(possibly still true) if you installed both Canopy and Anaconda, you
would break the first one because they tried to load the other's stdlib.
Other implementations have different structures or do not register
themselves at all, which also makes it more complicated for tools to
discover them.
So here is a rough proposal to standardise the registry keys that can be
set on Windows in a way that (a) lets other installers besides the
official ones have equal footing, (b) provides consistent search and
resolution semantics for tools, and (c) includes slightly more rich
metadata (such as display names and URLs). Presented in PEP-like form
here, but if feedback suggests just putting it in the docs I'm okay with
that too. It is fully backwards compatible with official releases of
Python (at least back to 2.5, possibly further) and does not require
modifications to Python or the official installer - it is purely
codifying a superset of what we already do.
Any and all feedback welcomed, especially from the owners of other
distros, Python implementations or tools on the list.
Cheers,
Steve
-----
PEP: ???
Title: Python environment registration in the Windows Registry
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower <steve...@python.org>
Status: Draft
Type: ???
Content-Type: text/x-rst
Created: 02-Feb-2016
Abstract
========
When installed on Windows, the official Python installer creates a
registry key for discovery and detection by other applications.
Unofficial installers, such as those used by distributions, typically
create identical keys for the same purpose. However, these may conflict
with the official installer or other distributions.
This PEP defines a schema for the Python registry key to allow
unofficial installers to separately register their installation, and to
allow applications to detect and correctly display all Python
environments on a user's machine. No implementation changes to Python
are proposed with this PEP.
The schema matches the registry values that have been used by the
official installer since at least Python 2.5, and the resolution
behaviour matches the behaviour of the official Python releases.
Specification
=============
We consider there to be a single collection of Python environments on a
machine, where the collection may be different for each user of the
machine. There are three potential registry locations where the
collection may be stored based on the installation options of each
environment. These are::
HKEY_CURRENT_USER\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag>
On a given machine, an environment is uniquely identified by its
Company-Tag pair. Keys should be searched in the order shown, and if the
same Company-Tag pair appears in more than one of the above locations,
only the first occurrence is offerred.
Official Python releases use ``PythonCore`` for Company, and the value
of ``sys.winver`` for Tag. Other registered environments may use any
values for Company and Tag. Recommendations are made in the following
sections.
Backwards Compatibility
-----------------------
Python 3.4 and earlier did not distinguish between 32-bit and 64-bit
builds in ``sys.winver``. As a result, it is possible to have valid
side-by-side installations of both 32-bit and 64-bit interpreters.
To ensure backwards compatibility, applications should treat
environments listed under the following two registry keys as distinct,
even if Tag matches::
HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>
Note that this does not apply to Python 3.5 and later, which uses
different Tags. Environments registered under other Company names must
use distinct Tags for side-by-side installations.
1. Environments in ``HKEY_CURRENT_USER`` are always preferred
2. Environments in ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` are
preferred if the interpreter is known to be 32-bit
Company
-------
The Company part of the key is intended to group related environments
and to ensure that Tags are namespaced appropriately. The key name
should be alphanumeric without spaces and likely to be unique. For
example, a trademarked name, a UUID, or a hostname would be appropriate::
HKEY_CURRENT_USER\Software\Python\ExampleCorp
HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60
HKEY_CURRENT_USER\Software\Python\www.example.com
If a string value named ``DisplayName`` exists, it should be used to
identify the environment category to users. Otherwise, the name of the
key should be used.
If a string value named ``SupportUrl`` exists, it may be displayed or
otherwise used to direct users to a web site related to the environment.
A complete example may look like::
HKEY_CURRENT_USER\Software\Python\ExampleCorp
(Default) = (value not set)
DisplayName = "Example Corp"
SupportUrl = "http://www.example.com"
Tag
---
The Tag part of the key is intended to uniquely identify an environment
within those provided by a single company. The key name should be
alphanumeric without spaces and stable across installations. For
example, the Python language version, a UUID or a partial/complete hash
would be appropriate; an integer counter that increases for each new
environment may not::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66
If a string value named ``DisplayName`` exists, it should be used to
identify the environment to users. Otherwise, the name of the key should
be used.
If a string value named ``SupportUrl`` exists, it may be displayed or
otherwise used to direct users to a web site related to the environment.
If a string value named ``Version`` exists, it should be used to
identify the version of the environment. This is independent from the
version of Python implemented by the environment.
If a string value named ``SysVersion`` exists, it must be in ``x.y`` or
``x.y.z`` format matching the version returned by ``sys.version_info``
in the interpreter. Otherwise, if the Tag matches this format it is
used. If not, the Python version is unknown.
Note that each of these values is recommended, but optional. A complete
example may look like this::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66
(Default) = (value not set)
DisplayName = "Distro 3"
SupportUrl = "http://www.example.com/distro-3"
Version = "3.0.12345.0"
SysVersion = "3.6.0"
InstallPath
-----------
Beneath the environment key, an ``InstallPath`` key must be created.
This key is always named ``InstallPath``, and the default value must
match ``sys.prefix``::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\3.6\InstallPath
(Default) = "C:\ExampleCorpPy36"
If a string value named ``ExecutablePath`` exists, it must be a path to
the ``python.exe`` (or equivalent) executable. Otherwise, the
interpreter executable is assumed to be called ``python.exe`` and exist
in the directory referenced by the default value.
If a string value named ``WindowedExecutablePath`` exists, it must be a
path to the ``pythonw.exe`` (or equivalent) executable. Otherwise, the
windowed interpreter executable is assumed to be called ``pythonw.exe``
and exist in the directory referenced by the default value.
A complete example may look like::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\InstallPath
(Default) = "C:\ExampleDistro30"
ExecutablePath = "C:\ExampleDistro30\ex_python.exe"
WindowedExecutablePath = "C:\ExampleDistro30\ex_pythonw.exe"
Help
----
Beneath the environment key, a ``Help`` key may be created. This key is
always named ``Help`` if present and has no default value.
Each subkey of ``Help`` specifies a documentation file, tool, or URL
associated with the environment. The subkey may have any name, and the
default value is a string appropriate for passing to ``os.startfile`` or
equivalent.
If a string value named ``DisplayName`` exists, it should be used to
identify the help file to users. Otherwise, the key name should be used.
A complete example may look like::
HKEY_CURRENT_USER\Software\Python\ExampleCorp\6C465E66\Help
Python\
(Default) = "C:\ExampleDistro30\python36.chm"
DisplayName = "Python Documentation"
Extras\
(Default) = "http://www.example.com/tutorial"
DisplayName = "Example Distro Online Tutorial"
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com
Alexander Walters
Feb 3, 2016, 3:16:57 AM2/3/16
to pytho...@python.org
...just when I thought I have solved the registry headaches I have been
dealing with...
I am not saying this proposal will make the registry situation worse,
but it may break my solution to the headaches Python's registry use
causes with some non-standard module installers (and even the standard
distutils exe installers, but that is being mitigated). In the wild
exist modules with their own EXE or MSI installers that check the
registry for 'the system python'. No matter how hard you hit them, they
will only install to *that one python*. You can imagine the sadist that
builds such an installer would not be receptive to the concept of
wheels. So in order to force those modules to install to the python YOU
want, dammit, you have to edit the registry. I have (contributed to) a
script that just sets whatever python it was last run with to the one
true system python. Works with virtualenvs too.This is not a terribly
obscure script either, by the way. It is in the first reply to the
google search for "how to install exes in a virtualenv".
So here I am in a situation where some pythons use another registry
entry. I have no idea if this makes my life as a user harder. What does
this kind of meddling do against tools trying to autodetect python, and
finding an ever changing value? Are we trying to guarantee that the
keys used by the python.org installers or only ever actually used by
standard CPython?
I know for PTVS manually adding a python environment to visual studio is
trivial - you fill in three locations, and its done. Just today I added
a python environment to my system that was not autodetected. It took
under a minute and almost no effort to add it... so for that tool this
adds very little benefit. I do not know about other tools.
On a very personal note (like the rest of this wasn't my personal issues
with possibly making my life slightly more difficult), I would much
rather see python stop touching the registry all together - but I have
no strong argument for that.
> https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com
dealing with...
I am not saying this proposal will make the registry situation worse,
but it may break my solution to the headaches Python's registry use
causes with some non-standard module installers (and even the standard
distutils exe installers, but that is being mitigated). In the wild
exist modules with their own EXE or MSI installers that check the
registry for 'the system python'. No matter how hard you hit them, they
will only install to *that one python*. You can imagine the sadist that
builds such an installer would not be receptive to the concept of
wheels. So in order to force those modules to install to the python YOU
want, dammit, you have to edit the registry. I have (contributed to) a
script that just sets whatever python it was last run with to the one
true system python. Works with virtualenvs too.This is not a terribly
obscure script either, by the way. It is in the first reply to the
google search for "how to install exes in a virtualenv".
So here I am in a situation where some pythons use another registry
entry. I have no idea if this makes my life as a user harder. What does
this kind of meddling do against tools trying to autodetect python, and
finding an ever changing value? Are we trying to guarantee that the
keys used by the python.org installers or only ever actually used by
standard CPython?
I know for PTVS manually adding a python environment to visual studio is
trivial - you fill in three locations, and its done. Just today I added
a python environment to my system that was not autodetected. It took
under a minute and almost no effort to add it... so for that tool this
adds very little benefit. I do not know about other tools.
On a very personal note (like the rest of this wasn't my personal issues
with possibly making my life slightly more difficult), I would much
rather see python stop touching the registry all together - but I have
no strong argument for that.
Glenn Linderman
Feb 3, 2016, 4:52:21 AM2/3/16
to pytho...@python.org
On 2/3/2016 12:15 AM, Alexander Walters
wrote:
On a very personal note (like the rest of this wasn't my personal issues with possibly making my life slightly more difficult), I would much rather see python stop touching the registry all together - but I have no strong argument for that.
Me too. My opinions follow, some might call them arguments, strong
or weak, some might call them foolishness.
I've been setting up a collection of tools and programs on Dropbox for far-flung project team members (users, not programmers) to share. It is nearly impossible to install a typical Windows program on Dropbox, because the installation information is partly in the installed directory structure, and partly in the registry. Exporting registry info from the machine that does the install to other machines is hard because other users have different paths to Dropbox. OK, for commercial software, installing to Dropbox probably violates some of the licensing, forcing each user to buy/install, but FOSS or in-house software doesn't have such restrictions: but still, most of it wants to use the registry, and Windows almost, but doesn't quite, force it to.
Portable software exists, but often is 3rd party hacks of popular FOSS rather than being directly supported by the FOSS development team. Python falls into this category. Happily, I recently found WinPython Zero, which hacks it (somehow) to work portably, and I've deployed it successfully on Dropbox. I'd rather Python were portable on its own, without hacks.
Portability requires not using the registry, so I agree with Alexander there.
Portability, as "Windows portable software" is generally defined, is focused on moving programs and data around on a flash drive, from one computer to another, and is focused on single-user, any (Windows) machine (with sufficient specs).
That doesn't quite fit the Dropbox environment. Most portable software reintroduces the idea of storing configuration information in the program folder, which is OK for "project" configuration info, done once, by one person, but not for "personal preferences".
The other thing Windows GUI lost is the concept of "current working directory", which hit me hard when I first started trying to set up project working areas on Dropbox. Many Windows programs want to run only one copy of themselves, in one Window, with one set of settings, and one "Start In" directory (which generally defaults to .... the program directory, or sometimes to "My Documents". This is why I went looking for a portable Python (and other portable things), and I finally realized that I was "fighting city hall" in trying to get an environment set up that was usable for the various teams (of users, not developers). Writeup at slashdot for more details on the lack of a "current working directory" concept in Windows GUI programs.
The path to Dropbox folders is different for everyone, even the drive letter can be different.
So here are my preferences for CPython:
1) It would be best CPython itself were fully portable. That wherever you point the installer, it lives there, and if somehow you happen to execute it from that directory, it would use its invocation path as the basis for finding the rest of the installed pieces.
2) A script could be provided that sets the association for .py and the corresponding ftype to point to the python that executed the script... and which has a parameter to clear that back out, too. This would be to allow users to set a particular python for convenient script execution, since Windows does the association thing, instead of using #! lines.
3) A script could be provided (maybe the one Alexander referred to) that sets the registry so that the "apparently one true System Python" points to the python that executed the script... and which has a parameter to clear that back out, too. This would be for compatible transition to the new registry-free Python versions for packages that have weird installers (such as Alexander alluded to). But with the registry-free Python available, those packages would hopefully see the future is registry free, and avoid requiring registry data to do installs.
4) A script could be provided to add the python that executed the script to the PATH, with an option to remove it.
5) A script could be provided to add the python that executed the script to the launcher .ini file, with an option to remove it.
6) A script could be provided to add the python that executed it to the Start menu, and/or Desktop icons, with an option to remove them.
Maybe scripts 2-6 are all the same one, with different options (or different invocation shortcuts for the clicky folk). Not everyone would probably need all the scripts, but none of them would be particularly large, such that they would be burdensome for others to ignore.
Such a collection of scripts would allow folks to achieve various levels of integration with Windows "conveniences", without requiring it. The portability would allow Python to be installed on Dropbox or a network share, and used from there without requiring every team member to do all the individual installs of the packages needed for a project.
I've been setting up a collection of tools and programs on Dropbox for far-flung project team members (users, not programmers) to share. It is nearly impossible to install a typical Windows program on Dropbox, because the installation information is partly in the installed directory structure, and partly in the registry. Exporting registry info from the machine that does the install to other machines is hard because other users have different paths to Dropbox. OK, for commercial software, installing to Dropbox probably violates some of the licensing, forcing each user to buy/install, but FOSS or in-house software doesn't have such restrictions: but still, most of it wants to use the registry, and Windows almost, but doesn't quite, force it to.
Portable software exists, but often is 3rd party hacks of popular FOSS rather than being directly supported by the FOSS development team. Python falls into this category. Happily, I recently found WinPython Zero, which hacks it (somehow) to work portably, and I've deployed it successfully on Dropbox. I'd rather Python were portable on its own, without hacks.
Portability requires not using the registry, so I agree with Alexander there.
Portability, as "Windows portable software" is generally defined, is focused on moving programs and data around on a flash drive, from one computer to another, and is focused on single-user, any (Windows) machine (with sufficient specs).
That doesn't quite fit the Dropbox environment. Most portable software reintroduces the idea of storing configuration information in the program folder, which is OK for "project" configuration info, done once, by one person, but not for "personal preferences".
The other thing Windows GUI lost is the concept of "current working directory", which hit me hard when I first started trying to set up project working areas on Dropbox. Many Windows programs want to run only one copy of themselves, in one Window, with one set of settings, and one "Start In" directory (which generally defaults to .... the program directory, or sometimes to "My Documents". This is why I went looking for a portable Python (and other portable things), and I finally realized that I was "fighting city hall" in trying to get an environment set up that was usable for the various teams (of users, not developers). Writeup at slashdot for more details on the lack of a "current working directory" concept in Windows GUI programs.
The path to Dropbox folders is different for everyone, even the drive letter can be different.
So here are my preferences for CPython:
1) It would be best CPython itself were fully portable. That wherever you point the installer, it lives there, and if somehow you happen to execute it from that directory, it would use its invocation path as the basis for finding the rest of the installed pieces.
2) A script could be provided that sets the association for .py and the corresponding ftype to point to the python that executed the script... and which has a parameter to clear that back out, too. This would be to allow users to set a particular python for convenient script execution, since Windows does the association thing, instead of using #! lines.
3) A script could be provided (maybe the one Alexander referred to) that sets the registry so that the "apparently one true System Python" points to the python that executed the script... and which has a parameter to clear that back out, too. This would be for compatible transition to the new registry-free Python versions for packages that have weird installers (such as Alexander alluded to). But with the registry-free Python available, those packages would hopefully see the future is registry free, and avoid requiring registry data to do installs.
4) A script could be provided to add the python that executed the script to the PATH, with an option to remove it.
5) A script could be provided to add the python that executed the script to the launcher .ini file, with an option to remove it.
6) A script could be provided to add the python that executed it to the Start menu, and/or Desktop icons, with an option to remove them.
Maybe scripts 2-6 are all the same one, with different options (or different invocation shortcuts for the clicky folk). Not everyone would probably need all the scripts, but none of them would be particularly large, such that they would be burdensome for others to ignore.
Such a collection of scripts would allow folks to achieve various levels of integration with Windows "conveniences", without requiring it. The portability would allow Python to be installed on Dropbox or a network share, and used from there without requiring every team member to do all the individual installs of the packages needed for a project.
Paul Moore
Feb 3, 2016, 6:22:45 AM2/3/16
to Glenn Linderman, Python Dev
On 3 February 2016 at 09:50, Glenn Linderman <v+py...@g.nevcal.com> wrote:
> On 2/3/2016 12:15 AM, Alexander Walters wrote:
>
>> On a very personal note (like the rest of this wasn't my personal issues
>> with possibly making my life slightly more difficult), I would much rather
>> see python stop touching the registry all together - but I have no strong
>> argument for that.
>
> Me too. My opinions follow, some might call them arguments, strong or weak,
> some might call them foolishness.
I also would prefer that Python not use/need the registry. As far as I
> On 2/3/2016 12:15 AM, Alexander Walters wrote:
>
>> On a very personal note (like the rest of this wasn't my personal issues
>> with possibly making my life slightly more difficult), I would much rather
>> see python stop touching the registry all together - but I have no strong
>> argument for that.
>
> Me too. My opinions follow, some might call them arguments, strong or weak,
> some might call them foolishness.
know, in practical terms Python works just fine without the registry
entries being present, but I'd like it if that were formally supported
as an option for users. Obviously it means that tools that want to see
what Python versions are installed won't see such unregistered copies,
I'm fine with that (and I don't have a problem with leaving it up to
said tools as to whether they want to include ways to support using a
Python installation that's not in the registry).
Some issues with this proposal:
1. I don't like the way it states that Python distributions "must" set
keys. I'd rather that it were explicitly stated that a distribution
which sets no registry keys is entirely valid (with the obvious
implication that tools which scan the registry looking for installed
Python distributions won't be able to see such an installation).
2. It's not clear to me how alternative distributions will specify
their registry keys (or the fact that they don't use them). The
registry keys are built into the Python interpreter - this proposal
seems to imply that distributions that simply repackage the python.org
build will no longer be valid or supported, and instead anyone who
wants to produce a custom Python distribution (even private in-house
repackagings) will need to change the Python source and rebuild.
That's a pretty major change, and if that *is* the intent, then as a
minimum I'd say we need to provide compiler flags to let
redistributors specify their Company/Tag values (or say that they want
to omit registry use). And I'm still not happy that "repackage the
python.org binaries" has been removed as an option.
It's possible that the reason the above two points have been missed is
because the proposal focuses purely on "informational" registry data.
But Python also modifies sys.path based on the registry entries - and
possibly has other behavioural changes as well. The pywin32 package,
in particular, makes use of this (it's a feature of pywin32 that I
disagree with and I wish it didn't do that, but it does, and it's a
very widely used package on Windows). So ignoring this aspect of
Python's behaviour is a big problem. (Also, what changes will pywin32
need to make to correctly support being installed into non-python.org
distributions when this proposal is implemented?)
Paul
Steve Dower
Feb 3, 2016, 11:19:59 AM2/3/16
to Alexander Walters, pytho...@python.org
On 03Feb2016 0015, Alexander Walters wrote:
> I am not saying this proposal will make the registry situation worse,
> but it may break my solution to the headaches Python's registry use
> causes with some non-standard module installers (and even the standard
> distutils exe installers, but that is being mitigated). In the wild
> exist modules with their own EXE or MSI installers that check the
> registry for 'the system python'. No matter how hard you hit them, they
> will only install to *that one python*. You can imagine the sadist that
> builds such an installer would not be receptive to the concept of
> wheels.
(I agree, but maybe if you can point them to a PEP it'll help? Right
> I am not saying this proposal will make the registry situation worse,
> but it may break my solution to the headaches Python's registry use
> causes with some non-standard module installers (and even the standard
> distutils exe installers, but that is being mitigated). In the wild
> exist modules with their own EXE or MSI installers that check the
> registry for 'the system python'. No matter how hard you hit them, they
> will only install to *that one python*. You can imagine the sadist that
> builds such an installer would not be receptive to the concept of
> wheels.
now, there's nowhere to even point them to.)
> So in order to force those modules to install to the python YOU
> want, dammit, you have to edit the registry. I have (contributed to) a
> script that just sets whatever python it was last run with to the one
> true system python. Works with virtualenvs too.This is not a terribly
> obscure script either, by the way. It is in the first reply to the
> google search for "how to install exes in a virtualenv".
post-update will still look in PythonCore\X.Y for Python.
> So here I am in a situation where some pythons use another registry
> entry. I have no idea if this makes my life as a user harder. What does
> this kind of meddling do against tools trying to autodetect python, and
> finding an ever changing value? Are we trying to guarantee that the
> keys used by the python.org installers or only ever actually used by
> standard CPython?
but the other part is officially telling everyone else that they are
welcome to create their own keys.
> I know for PTVS manually adding a python environment to visual studio is
> trivial - you fill in three locations, and its done. Just today I added
> a python environment to my system that was not autodetected. It took
> under a minute and almost no effort to add it... so for that tool this
> adds very little benefit. I do not know about other tools.
those three locations :) But I don't think people should need to do that
by hand at all.
For example, the path to an Anaconda installation is buried deep inside
AppData (as is Python 3.5+ now), and varies based on your username.
Canopy does the same, and once you've found it there are (or were?) at
least three copies of python.exe to choose from (we worked with
Enthought to make this Just Work for PTVS users).
> On a very personal note (like the rest of this wasn't my personal issues
> with possibly making my life slightly more difficult), I would much
> rather see python stop touching the registry all together - but I have
> no strong argument for that.
configuration store on Windows, and when we need global state it is the
place to go. (My actual hope for Python 3.6 is to drop the few places
where Python *reads* from the registry for configuration that shouldn't
be global, at which point the Python key is solely for other programs.
But those fixes are probably not PEP-worthy.)
Cheers,
Steve
Steve Dower
Feb 3, 2016, 11:31:39 AM2/3/16
to Glenn Linderman, pytho...@python.org
On 03Feb2016 0150, Glenn Linderman wrote:
> Portable software exists, but often is 3rd party hacks of popular FOSS
> rather than being directly supported by the FOSS development team.
> Python falls into this category. Happily, I recently found WinPython
> Zero, which hacks it (somehow) to work portably, and I've deployed it
> successfully on Dropbox. I'd rather Python were portable on its own,
> without hacks.
>
> Portability requires not using the registry, so I agree with Alexander
> there.
Python has been registry independent for a while if you set the
> Portable software exists, but often is 3rd party hacks of popular FOSS
> rather than being directly supported by the FOSS development team.
> Python falls into this category. Happily, I recently found WinPython
> Zero, which hacks it (somehow) to work portably, and I've deployed it
> successfully on Dropbox. I'd rather Python were portable on its own,
> without hacks.
>
> Portability requires not using the registry, so I agree with Alexander
> there.
PYTHONHOME environment variable (and independent but potentially
unreliable even without this).
Most of the registry settings created on install are for supporting
upgrades, repairs and uninstallation, none of which matter in your case.
(Also many python-dev readers' cases, but there are a lot of users who
get into trouble very quickly without this level of management.)
> So here are my preferences for CPython:
>
> 1) It would be best CPython itself were fully portable. That wherever
> you point the installer, it lives there, and if somehow you happen to
> execute it from that directory, it would use its invocation path as the
> basis for finding the rest of the installed pieces.
> 2) A script could be provided that sets the association for .py and the
> corresponding ftype to point to the python that executed the script...
> and which has a parameter to clear that back out, too. This would be to
> allow users to set a particular python for convenient script execution,
> since Windows does the association thing, instead of using #! lines.
being registered...), as it handles shebangs - even /usr/bin/env style.
A script such as what you're asking for would be possible, but not
something I want to be responsible for providing and maintaining.
> 3) A script could be provided (maybe the one Alexander referred to) that
> sets the registry so that the "apparently one true System Python" points
> to the python that executed the script... and which has a parameter to
> clear that back out, too. This would be for compatible transition to
> the new registry-free Python versions for packages that have weird
> installers (such as Alexander alluded to). But with the registry-free
> Python available, those packages would hopefully see the future is
> registry free, and avoid requiring registry data to do installs.
> 4) A script could be provided to add the python that executed the script
> to the PATH, with an option to remove it.
>
> 5) A script could be provided to add the python that executed the script
> to the launcher .ini file, with an option to remove it.
>
> 6) A script could be provided to add the python that executed it to the
> Start menu, and/or Desktop icons, with an option to remove them.
are also able to do this.
> Maybe scripts 2-6 are all the same one, with different options (or
> different invocation shortcuts for the clicky folk). Not everyone would
> probably need all the scripts, but none of them would be particularly
> large, such that they would be burdensome for others to ignore.
>
> Such a collection of scripts would allow folks to achieve various levels
> of integration with Windows "conveniences", without requiring it. The
> portability would allow Python to be installed on Dropbox or a network
> share, and used from there without requiring every team member to do all
> the individual installs of the packages needed for a project.
Final point I want to reiterate - Python itself is essentially registry
free already in that it does not need registry settings to function. The
current problems are:
1. other programs need to locate all available Pythons
2. there appears to only be one space to register your Python
3. this space is *sometimes* used by Python to locate itself and
installed packages
I want to fix problem 2 via documentation, and then look at the much
more difficult problem 3.
Cheers,
Steve
Paul Moore
Feb 3, 2016, 11:40:13 AM2/3/16
to Steve Dower, Python Dev
On 3 February 2016 at 16:29, Steve Dower <steve...@python.org> wrote:
> Final point I want to reiterate - Python itself is essentially registry free
> already in that it does not need registry settings to function.
That's something we should probably publicise better. People seem
> Final point I want to reiterate - Python itself is essentially registry free
> already in that it does not need registry settings to function.
unaware of it (in much the same way that they never really noticed zip
application support). Maybe we could include a section in the Python
3.6 "What's new" (even though it's not technically new - but I did a
quick check of What's New back to 3.2 and couldn't see any mention)?
Paul
Steve Dower
Feb 3, 2016, 11:49:02 AM2/3/16
to Paul Moore, Glenn Linderman, Python Dev
On 03Feb2016 0321, Paul Moore wrote:
> Some issues with this proposal:
>
> 1. I don't like the way it states that Python distributions "must" set
> keys. I'd rather that it were explicitly stated that a distribution
> which sets no registry keys is entirely valid (with the obvious
> implication that tools which scan the registry looking for installed
> Python distributions won't be able to see such an installation).
Good point, I never meant to imply that. If you don't want your Python install/env found then you don't have to register anything.
> Some issues with this proposal:
>
> 1. I don't like the way it states that Python distributions "must" set
> keys. I'd rather that it were explicitly stated that a distribution
> which sets no registry keys is entirely valid (with the obvious
> implication that tools which scan the registry looking for installed
> Python distributions won't be able to see such an installation).
(Of course, when users come to us IDE developers and say "your tool can't find Python XYZ", we'll all just go to Python XYZ and say "your users need you to register when you install, see this PEP" :) )
> 2. It's not clear to me how alternative distributions will specify
> their registry keys (or the fact that they don't use them). The
> registry keys are built into the Python interpreter - this proposal
> seems to imply that distributions that simply repackage the python.org
> build will no longer be valid or supported, and instead anyone who
> wants to produce a custom Python distribution (even private in-house
> repackagings) will need to change the Python source and rebuild.
> That's a pretty major change, and if that *is* the intent, then as a
> minimum I'd say we need to provide compiler flags to let
> redistributors specify their Company/Tag values (or say that they want
> to omit registry use). And I'm still not happy that "repackage the
> python.org binaries" has been removed as an option.
sys.path = []
sys.path.append('')
sys.path.extend(os.getenv('PYTHONPATH').split(';'))
sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
home = os.getenv('PYTHONHOME')
if not home:
if os.path.exists(os.path.join(sys.argv[0], '..', 'Lib', 'os.py')):
home = os.path.dirname(sys.argv[0])
if not home:
paths = read_value(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath')
if not paths:
paths = read_value(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath')
if paths:
sys.path.extend(paths.split(';'))
else:
sys.path.append(r'.\Lib')
# more well-known subdirs
else:
sys.path.append(os.path.join(home, 'Lib'))
# more well-known subdirs ...
So a few high-level observations:
* any program can install anywhere on the machine and make its libraries available to a specific version of Python by creating a subkey under 'PythonCore\x.y\PythonPath'
* any environment lacking 'Lib\os.py' (e.g. venv) relies on the registry to locate enough stdlib to import site
* this is too complicated, but guaranteed we will break users in production if we change it now
So if repackagers follow a few rules (that I documented in https://docs.python.org/3.5/using/windows.html - I see the process above is also documented there, which I wish I remembered before writing all that out), they'll be fine. Unfortunately, following those rules means that you don't register anywhere that separate tools can find you, and so users complain and you "fix" it by doing the wrong thing.
This PEP offers a right way to fix it.
> It's possible that the reason the above two points have been missed is
> because the proposal focuses purely on "informational" registry data.
> But Python also modifies sys.path based on the registry entries - and
> possibly has other behavioural changes as well. The pywin32 package,
> in particular, makes use of this (it's a feature of pywin32 that I
> disagree with and I wish it didn't do that, but it does, and it's a
> very widely used package on Windows). So ignoring this aspect of
> Python's behaviour is a big problem. (Also, what changes will pywin32
> need to make to correctly support being installed into non-python.org
> distributions when this proposal is implemented?)
Cheers,
Steve
Steve Dower
Feb 3, 2016, 12:06:28 PM2/3/16
to Paul Moore, Python Dev
On 03Feb2016 0839, Paul Moore wrote:
> On 3 February 2016 at 16:29, Steve Dower <steve...@python.org> wrote:
>> Final point I want to reiterate - Python itself is essentially registry free
>> already in that it does not need registry settings to function.
>
> That's something we should probably publicise better. People seem
> unaware of it (in much the same way that they never really noticed zip
> application support). Maybe we could include a section in the Python
> 3.6 "What's new" (even though it's not technically new - but I did a
> quick check of What's New back to 3.2 and couldn't see any mention)?
Maybe, but since it is still potentially problematic I'd rather not
> On 3 February 2016 at 16:29, Steve Dower <steve...@python.org> wrote:
>> Final point I want to reiterate - Python itself is essentially registry free
>> already in that it does not need registry settings to function.
>
> That's something we should probably publicise better. People seem
> unaware of it (in much the same way that they never really noticed zip
> application support). Maybe we could include a section in the Python
> 3.6 "What's new" (even though it's not technically new - but I did a
> quick check of What's New back to 3.2 and couldn't see any mention)?
right now.
Basically, I don't want to have to support people whose "portable"
version of Python works fine on one machine, but has syntax errors in
the stdlib on another machine.
Adding the applocal option for 3.5 (described at
https://docs.python.org/3.5/using/windows.html) helps with this, but I'm
guessing always running in isolated mode is not what most people really
want.
Until we genuinely never rely on the registry, I don't want to claim
that we don't rely on the registry.
Cheers,
Steve
---
Rest of the email is spelling out how to create the scenario above,
since I assume people won't believe it :)
1. Take Python 3.4.1, install it (Just for Me), zip up the stdlib into
python34.zip and copy the binaries and zip to a "portable" folder
2. Update to Python 3.4.2 on the main machine
3. Run "-m test test_grammar" with your portable 3.4.1.
4. Boom! SyntaxError in test_grammar.py because you picked up the 3.4.2
stdlib (the error comes from https://hg.python.org/cpython/rev/4ad33d82193d)
Paul Moore
Feb 3, 2016, 12:13:52 PM2/3/16
to Steve Dower, Python Dev
On 3 February 2016 at 16:46, Steve Dower <steve...@python.org> wrote:
> So a few high-level observations:
>
> * any program can install anywhere on the machine and make its libraries available to a specific version of Python by creating a subkey under 'PythonCore\x.y\PythonPath'
Yeah, that's horrid but not really something we can change without
> So a few high-level observations:
>
> * any program can install anywhere on the machine and make its libraries available to a specific version of Python by creating a subkey under 'PythonCore\x.y\PythonPath'
compatibility breakage. But it's essentially opt-in - if you don't
need the feature you don't have to register anything under that key.
> * any environment lacking 'Lib\os.py' (e.g. venv) relies on the registry to locate enough stdlib to import site
> * this is too complicated, but guaranteed we will break users in production if we change it now
>
> So if repackagers follow a few rules (that I documented in https://docs.python.org/3.5/using/windows.html - I see the process above is also documented there, which I wish I remembered before writing all that out), they'll be fine. Unfortunately, following those rules means that you don't register anywhere that separate tools can find you, and so users complain and you "fix" it by doing the wrong thing.
for the documentation, failed to find it and then whined about it not
being documented - my apologies!)
More specifically for the people wanting "portable" Python systems, if
I read that right then if python.exe is alongside a Lib directory
containing os.py, Python needs no environment variables, and no
registry entries, to run perfectly. I don't see any issue with Python
builds that don't register themselves not showing up in tools that
look for Python installations. And as you say, if we give people who
make "official" distributions a way to properly register, then that
helps them and leaves the unregistered case for "homebrew" portable
copies of Python.
(I've just seen your other note about it being "potentially
problematic". OK, let's leave it low-key for now, but when we are
comfortable with it, can we publicise it then? I get a definite
impression that quite a lot of people assume that "you can't have a
portable build of Python").
> This PEP offers a right way to fix it.
So +0.5 from me for this PEP (Only 0.5, because I still have some
concerns that talking about registry entries in such detail gives the
impression that Python is tied to them more than it actually is. If
you can see a way of toning down the wording, then great, but better
to document the proposal accurately than to water it down because
people might get a mistaken impression).
>> It's possible that the reason the above two points have been missed is
>> because the proposal focuses purely on "informational" registry data.
>> But Python also modifies sys.path based on the registry entries - and
>> possibly has other behavioural changes as well. The pywin32 package,
>> in particular, makes use of this (it's a feature of pywin32 that I
>> disagree with and I wish it didn't do that, but it does, and it's a
>> very widely used package on Windows). So ignoring this aspect of
>> Python's behaviour is a big problem. (Also, what changes will pywin32
>> need to make to correctly support being installed into non-python.org
>> distributions when this proposal is implemented?)
>
> I haven't looked into pywin32's use of this recently - I tend to only use Christoph Gohlke's wheels that don't register anything. But it is certainly a valid concern. Hopefully Mark Hammond is watching :)
since I really used it. Like you, I go for wheels only these days.
Paul Moore
Feb 3, 2016, 12:16:36 PM2/3/16
to Steve Dower, Python Dev
On 3 February 2016 at 17:04, Steve Dower <steve...@python.org> wrote:
> Rest of the email is spelling out how to create the scenario above, since I
> assume people won't believe it :)
>
> 1. Take Python 3.4.1, install it (Just for Me), zip up the stdlib into
> python34.zip and copy the binaries and zip to a "portable" folder
>
> 2. Update to Python 3.4.2 on the main machine
>
> 3. Run "-m test test_grammar" with your portable 3.4.1.
>
> 4. Boom! SyntaxError in test_grammar.py because you picked up the 3.4.2
> stdlib (the error comes from https://hg.python.org/cpython/rev/4ad33d82193d)
Sigh. There's nothing so small that it isn't a compatibility break :-)
> Rest of the email is spelling out how to create the scenario above, since I
> assume people won't believe it :)
>
> 1. Take Python 3.4.1, install it (Just for Me), zip up the stdlib into
> python34.zip and copy the binaries and zip to a "portable" folder
>
> 2. Update to Python 3.4.2 on the main machine
>
> 3. Run "-m test test_grammar" with your portable 3.4.1.
>
> 4. Boom! SyntaxError in test_grammar.py because you picked up the 3.4.2
> stdlib (the error comes from https://hg.python.org/cpython/rev/4ad33d82193d)
But of course this process violates the rule "set PYTHONHOME or have a
Lib/os.py *file* alongside python.exe". Like you say the rules are
subtle enough that people will make mistakes :-(
Thanks for the explanation.
Paul
eryk sun
Feb 3, 2016, 12:25:15 PM2/3/16
to Python Dev
On Wed, Feb 3, 2016 at 10:46 AM, Steve Dower <steve...@python.org> wrote:
>
> sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
> sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
It seems like a bug (in spirit at least) that this step isn't skipped
>
> sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
> sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
for -E and -I (Py_IgnoreEnvironmentFlag, Py_IsolatedFlag).
> I haven't looked into pywin32's use of this recently - I tend to only use
> Christoph Gohlke's wheels that don't register anything.
# .pth file for the PyWin32 extensions
win32
win32\lib
Pythonwin
import os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pypiwin32_system32"))
This is different from a PythonPath subkey in a couple of respects.
The paths listed in .pth files are appended to sys.path instead of
prepended. They also don't get added when run with -S or for a venv
environment that excludes site-packages.
Steve Dower
Feb 3, 2016, 12:55:39 PM2/3/16
to eryk sun, Python Dev
On 03Feb2016 0923, eryk sun wrote:
> On Wed, Feb 3, 2016 at 10:46 AM, Steve Dower <steve...@python.org> wrote:
>>
>> sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
>> sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
>
> It seems like a bug (in spirit at least) that this step isn't skipped
> for -E and -I (Py_IgnoreEnvironmentFlag, Py_IsolatedFlag).
They should be skipped. If not, I'm very much in favour of fixing that
> On Wed, Feb 3, 2016 at 10:46 AM, Steve Dower <steve...@python.org> wrote:
>>
>> sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
>> sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
>
> It seems like a bug (in spirit at least) that this step isn't skipped
> for -E and -I (Py_IgnoreEnvironmentFlag, Py_IsolatedFlag).
immediately in all active branches.
>> I haven't looked into pywin32's use of this recently - I tend to only use
>> Christoph Gohlke's wheels that don't register anything.
>
> I install the pypiwin32 wheel using pip, which uses pypiwin32.pth:
>
> # .pth file for the PyWin32 extensions
> win32
> win32\lib
> Pythonwin
>
> import os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pypiwin32_system32"))
>
> This is different from a PythonPath subkey in a couple of respects.
> The paths listed in .pth files are appended to sys.path instead of
> prepended. They also don't get added when run with -S or for a venv
> environment that excludes site-packages.
files. However, this is not directly affected by the registry, so
thankfully not a concern right now.
Cheers,
Steve
Alexander Walters
Feb 3, 2016, 2:21:49 PM2/3/16
to pytho...@python.org
On 2/3/2016 11:17, Steve Dower wrote:
>> I know for PTVS manually adding a python environment to visual studio is
>> trivial - you fill in three locations, and its done. Just today I added
>> a python environment to my system that was not autodetected. It took
>> under a minute and almost no effort to add it... so for that tool this
>> adds very little benefit. I do not know about other tools.
>
> I'm also a PTVS maintainer, so I know how much magic is going on
> behind those three locations :) But I don't think people should need
> to do that by hand at all.
>
> For example, the path to an Anaconda installation is buried deep
> inside AppData (as is Python 3.5+ now), and varies based on your
> username. Canopy does the same, and once you've found it there are (or
> were?) at least three copies of python.exe to choose from (we worked
> with Enthought to make this Just Work for PTVS users).
privileges to edit the registry... (It wont ask to elevate unless you
install for all users, and that's where all users will install). So on
that point alone, this saves nothing substantive really. (I will go off
on python35 installing in insane locations some other time.)
Paul Moore
Feb 3, 2016, 2:33:58 PM2/3/16
to Alexander Walters, Python Dev
On 3 February 2016 at 19:20, Alexander Walters <tritiu...@sdamon.com> wrote:
> Uh.... its C:\Anaconda[2]\ for anyone running the installer with the
> privileges to edit the registry... (It wont ask to elevate unless you
> install for all users, and that's where all users will install). So on that
> point alone, this saves nothing substantive really.
Per-user installs go into HKCU, which doesn't require elevation, so
> Uh.... its C:\Anaconda[2]\ for anyone running the installer with the
> privileges to edit the registry... (It wont ask to elevate unless you
> install for all users, and that's where all users will install). So on that
> point alone, this saves nothing substantive really.
the proposal *does* help for per-user installs. And does the all-users
install really offer no option for users to choose their own install
location? Not even to switch to another drive (if, for example, C: is
protected)?
Paul
Matthew Einhorn
Feb 3, 2016, 3:03:49 PM2/3/16
to Python Development List
On Wed, Feb 3, 2016 at 3:15 AM, Alexander Walters <tritiu...@sdamon.com> wrote:
...just when I thought I have solved the registry headaches I have been dealing with...
I am not saying this proposal will make the registry situation worse, but it may break my solution to the headaches Python's registry use causes with some non-standard module installers (and even the standard distutils exe installers, but that is being mitigated). In the wild exist modules with their own EXE or MSI installers that check the registry for 'the system python'. No matter how hard you hit them, they will only install to *that one python*.
If I remember correctly, you can use `wheel convert filename.exe` on those installers which create a wheel that you can install. I think that's what I used to do with pywin32 before pypiwin32 came along.
I just tested it and it still works on the pywin32 exe.
I just tested it and it still works on the pywin32 exe.
Steve Dower
Feb 3, 2016, 6:47:26 PM2/3/16
to Alexander Walters, pytho...@python.org
On 03Feb2016 1120, Alexander Walters wrote:
> Uh.... its C:\Anaconda[2]\ for anyone running the installer with the
> privileges to edit the registry... (It wont ask to elevate unless you
> install for all users, and that's where all users will install). So on
> that point alone, this saves nothing substantive really. (I will go off
> on python35 installing in insane locations some other time.)
The install location is customisable, and users can always write to
> Uh.... its C:\Anaconda[2]\ for anyone running the installer with the
> privileges to edit the registry... (It wont ask to elevate unless you
> install for all users, and that's where all users will install). So on
> that point alone, this saves nothing substantive really. (I will go off
> on python35 installing in insane locations some other time.)
their own registry hive.
The same applies to Python, so you can choose to install it as
conveniently or as securely as you like. In either case, other
applications need a guaranteed place to find these installations, and
that place is the system registry.
Cheers,
Steve
Eric Snow
Feb 3, 2016, 8:34:23 PM2/3/16
to Steve Dower, Python-Dev
On Tue, Feb 2, 2016 at 10:15 PM, Steve Dower <steve...@python.org> wrote:
> I was throwing around some ideas with colleagues about how we detect Python
> installations on Windows from within Visual Studio, and it came up that
> there are many Python distros that install into different locations but
> write the same registry entries. (I knew about this, of course, but this
> time I decided to do something.)
>
> [snip]> I was throwing around some ideas with colleagues about how we detect Python
> installations on Windows from within Visual Studio, and it came up that
> there are many Python distros that install into different locations but
> write the same registry entries. (I knew about this, of course, but this
> time I decided to do something.)
>
>
> So here is a rough proposal to standardise the registry keys that can be set
> on Windows in a way that (a) lets other installers besides the official ones
> have equal footing, (b) provides consistent search and resolution semantics
> for tools, and (c) includes slightly more rich metadata (such as display
> names and URLs). Presented in PEP-like form here, but if feedback suggests
> just putting it in the docs I'm okay with that too. It is fully backwards
> compatible with official releases of Python (at least back to 2.5, possibly
> further) and does not require modifications to Python or the official
> installer - it is purely codifying a superset of what we already do.
>
> Any and all feedback welcomed, especially from the owners of other distros,
> Python implementations or tools on the list.
Just wanted to quickly point out another use of the WIndows registry
in Python: WindowsRegistryFinder [1]. This is an import "meta-path"
finder that locates modules declared (*not* defined) in the registry.
I'm not familiar with the Windows registry nor do I know if anyone is
using this finder. That said, ISTM the finder's use of the registry
does not face quite the same challenges you've described in the
proposal. I expect Martin von Löwis could explain more as he was
involved with adding the finder.
Just wanted to throw that out there, particularly if there's a chance
of the finder's registry keys conflicting in some way.
-eric
[1] https://hg.python.org/cpython/file/5873cfb42ebe/Lib/importlib/_bootstrap_external.py#l570
eryk sun
Feb 3, 2016, 9:52:47 PM2/3/16
to Python-Dev
On Wed, Feb 3, 2016 at 7:33 PM, Eric Snow <ericsnow...@gmail.com> wrote:
> Just wanted to quickly point out another use of the WIndows registry
> in Python: WindowsRegistryFinder [1]. This is an import "meta-path"
> finder that locates modules declared (*not* defined) in the registry.
> I'm not familiar with the Windows registry nor do I know if anyone is
> using this finder.
The "Modules" key (WindowsRegistryFinder in 3.3+ and previously
> Just wanted to quickly point out another use of the WIndows registry
> in Python: WindowsRegistryFinder [1]. This is an import "meta-path"
> finder that locates modules declared (*not* defined) in the registry.
> I'm not familiar with the Windows registry nor do I know if anyone is
> using this finder.
PyWin_FindRegisteredModule) adds individual modules by subkey name,
with the filepath in the default value (the filename can differ, but
it can't use an arbitrary extension). The "PythonPath" and "Modules"
keys both date back to Mark Hammond's Windows port in the mid 1990s.
Steve Dower
Feb 3, 2016, 10:01:22 PM2/3/16
to pytho...@python.org
On 03Feb2016 1851, eryk sun wrote:
> On Wed, Feb 3, 2016 at 7:33 PM, Eric Snow <ericsnow...@gmail.com> wrote:
>> Just wanted to quickly point out another use of the WIndows registry
>> in Python: WindowsRegistryFinder [1]. This is an import "meta-path"
>> finder that locates modules declared (*not* defined) in the registry.
>> I'm not familiar with the Windows registry nor do I know if anyone is
>> using this finder.
>
> The "Modules" key (WindowsRegistryFinder in 3.3+ and previously
> PyWin_FindRegisteredModule) adds individual modules by subkey name,
> with the filepath in the default value (the filename can differ, but
> it can't use an arbitrary extension). The "PythonPath" and "Modules"
> keys both date back to Mark Hammond's Windows port in the mid 1990s.
Yep, essentially, I expect these keys that actually affect how Python
> On Wed, Feb 3, 2016 at 7:33 PM, Eric Snow <ericsnow...@gmail.com> wrote:
>> Just wanted to quickly point out another use of the WIndows registry
>> in Python: WindowsRegistryFinder [1]. This is an import "meta-path"
>> finder that locates modules declared (*not* defined) in the registry.
>> I'm not familiar with the Windows registry nor do I know if anyone is
>> using this finder.
>
> The "Modules" key (WindowsRegistryFinder in 3.3+ and previously
> PyWin_FindRegisteredModule) adds individual modules by subkey name,
> with the filepath in the default value (the filename can differ, but
> it can't use an arbitrary extension). The "PythonPath" and "Modules"
> keys both date back to Mark Hammond's Windows port in the mid 1990s.
works to remain under PythonCore, and continue not to be documented or
recommended for general use. But I see no reason to deprecate or remove
them.
Specialised situations that use these keys should continue to set them
under PythonCore. I hope that is sufficiently implied by saying nothing
about them in the PEP - I really don't want to have to be more explicit
about it and I definitely do not want to actually name or list them in
any way.
Cheers,
Steve
Nick Coghlan
Feb 4, 2016, 8:52:56 AM2/4/16
to Steve Dower, pytho...@python.org
On 3 February 2016 at 15:15, Steve Dower <steve...@python.org> wrote:
> Presented in PEP-like form here, but if feedback suggests
> just putting it in the docs I'm okay with that too.
We don't really have anywhere in the docs to track platform
> Presented in PEP-like form here, but if feedback suggests
> just putting it in the docs I'm okay with that too.
integration topics like this, so an Informational PEP is your best
bet.
Cheers,
Nick.
P.S. While I guess you *could* try to figure out a suitable home in
the docs, I don't think you'd gain anything particularly useful from
the additional effort
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Alexander Walters
Feb 4, 2016, 1:54:33 PM2/4/16
to pytho...@python.org
I am well aware of this. In the SO question I referenced, being the
first google hit related this this... that is the answer *I* gave.
It only works, in my experience, 60% of the time, and not with two
biggie packages (pywin32, for which you have to go to third parties
to get the wheel, which do not include all of pywin32, and wxpython)
and perhaps more.
Steve Dower
Feb 6, 2016, 4:02:39 PM2/6/16
to pytho...@python.org
I've posted an updated version of this PEP that should soon be visible at https://www.python.org/dev/peps/pep-0514.
Leaving aside the fact that the current implementation of Python relies on *other* information in the registry (that is not specified in this PEP), I'm still looking for feedback or concerns from developers who are likely to create or use the keys that are described here.
----------------
PEP: 514
Title: Python registration in the Windows registry
Type: Informational
Content-Type: text/x-rst
Created: 02-Feb-2016
Post-History: 02-Feb-2016
Abstract
========
This PEP defines a schema for the Python registry key to allow third-party
installers to register their installation, and to allow applications to detect
automatically discoverable by external tools.
The schema matches the registry values that have been used by the official
installer since at least Python 2.5, and the resolution behaviour matches the
behaviour of the official Python releases.
Motivation
==========
When installed on Windows, the official Python installer creates a registry key
for discovery and detection by other applications. This allows tools such as
installers or IDEs to automatically detect and display a user's Python
installations.
Third-party installers, such as those used by distributions, typically create
identical keys for the same purpose. Most tools that use the registry to detect
Python installations only inspect the keys used by the official installer. As a
result, third-party installations that wish to be discoverable will overwrite
these values, resulting in users "losing" their Python installation.
By describing a layout for registry keys that allows third-party installations
to register themselves uniquely, as well as providing tool developers guidance
for discovering all available Python installations, these collisions should be
prevented.
Definitions
===========
A "registry key" is the equivalent of a file-system path into the registry. Each
key may contain "subkeys" (keys nested within keys) and "values" (named and
typed attributes attached to a key).
``HKEY_CURRENT_USER`` is the root of settings for the currently logged-in user,
and this user can generally read and write all settings under this root.
``HKEY_LOCAL_MACHINE`` is the root of settings for all users. Generally, any
user can read these settings but only administrators can modify them. It is
typical for values under ``HKEY_CURRENT_USER`` to take precedence over those in
``HKEY_LOCAL_MACHINE``.
On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key
that 32-bit processes transparently read and write to rather than accessing the
``Software`` key directly.
Structure
HKEY_CURRENT_USER\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag>
Environments are uniquely identified by their Company-Tag pair, with two options
for conflict resolution: include everything, or give priority to user
preferences.
Tools that include every installed environment, even where the Company-Tag pairs
match, should ensure users can easily identify whether the registration was
per-user or per-machine.
Tools that give priority to user preferences must ignore values from
``HKEY_LOCAL_MACHINE`` when a matching Company-Tag pair exists is in
``HKEY_CURRENT_USER``.
Official Python releases use ``PythonCore`` for Company, and the value of
``sys.winver`` for Tag. Other registered environments may use any values for
Company and Tag. Recommendations are made in the following sections.
Python environments are not required to register themselves unless they want to
be automatically discoverable by external tools.
Backwards Compatibility
-----------------------
Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in
``sys.winver``. As a result, it is possible to have valid side-by-side
installations of both 32-bit and 64-bit interpreters.
To ensure backwards compatibility, applications should treat environments listed
under the following two registry keys as distinct, even when the Tag matches::
HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>
Environments listed under ``HKEY_CURRENT_USER`` may be treated as distinct from
both of the above keys, potentially resulting in three environments discovered
using the same Tag. Alternatively, a tool may determine whether the per-user
environment is 64-bit or 32-bit and give it priority over the per-machine
environment, resulting in a maximum of two discovered environments.
It is not possible to detect side-by-side installations of both 64-bit and
32-bit versions of Python prior to 3.5 when they have been installed for the
current user. Python 3.5 and later always uses different Tags for 64-bit and
32-bit versions.
Environments registered under other Company names must use distinct Tags to
support side-by-side installations. There is no backwards compatibility
allowance.
Company
-------
The Company part of the key is intended to group related environments and to
ensure that Tags are namespaced appropriately. The key name should be
alphanumeric without spaces and likely to be unique. For example, a trademarked
name, a UUID, or a hostname would be appropriate::
HKEY_CURRENT_USER\Software\Python\ExampleCorp
HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60
HKEY_CURRENT_USER\Software\Python\www.example.com
The company name ``PyLauncher`` is reserved for the PEP 397 launcher
(``py.exe``). It does not follow this convention and should be ignored by tools.
Other Keys
----------
Some other registry keys are used for defining or inferring search paths under
certain conditions. A third-party installation is permitted to define these keys
under their Company-Tag key, however, the interpreter must be modified and
rebuilt in order to read these values.
Copyright
=========
This document has been placed in the public domain.
Leaving aside the fact that the current implementation of Python relies on *other* information in the registry (that is not specified in this PEP), I'm still looking for feedback or concerns from developers who are likely to create or use the keys that are described here.
----------------
PEP: 514
Title: Python registration in the Windows registry
Type: Informational
Content-Type: text/x-rst
Created: 02-Feb-2016
Post-History: 02-Feb-2016
Abstract
========
This PEP defines a schema for the Python registry key to allow third-party
installers to register their installation, and to allow applications to detect
and correctly display all Python environments on a user's machine. No
implementation changes to Python are proposed with this PEP.
Python environments are not required to be registered unless they want to be
implementation changes to Python are proposed with this PEP.
automatically discoverable by external tools.
The schema matches the registry values that have been used by the official
installer since at least Python 2.5, and the resolution behaviour matches the
behaviour of the official Python releases.
==========
When installed on Windows, the official Python installer creates a registry key
installers or IDEs to automatically detect and display a user's Python
installations.
Third-party installers, such as those used by distributions, typically create
identical keys for the same purpose. Most tools that use the registry to detect
Python installations only inspect the keys used by the official installer. As a
result, third-party installations that wish to be discoverable will overwrite
these values, resulting in users "losing" their Python installation.
By describing a layout for registry keys that allows third-party installations
to register themselves uniquely, as well as providing tool developers guidance
for discovering all available Python installations, these collisions should be
prevented.
Definitions
===========
A "registry key" is the equivalent of a file-system path into the registry. Each
key may contain "subkeys" (keys nested within keys) and "values" (named and
typed attributes attached to a key).
``HKEY_CURRENT_USER`` is the root of settings for the currently logged-in user,
and this user can generally read and write all settings under this root.
``HKEY_LOCAL_MACHINE`` is the root of settings for all users. Generally, any
user can read these settings but only administrators can modify them. It is
typical for values under ``HKEY_CURRENT_USER`` to take precedence over those in
``HKEY_LOCAL_MACHINE``.
On 64-bit Windows, ``HKEY_LOCAL_MACHINE\Software\Wow6432Node`` is a special key
that 32-bit processes transparently read and write to rather than accessing the
``Software`` key directly.
Structure
=========
We consider there to be a single collection of Python environments on a machine,
where the collection may be different for each user of the machine. There are
three potential registry locations where the collection may be stored based on
the installation options of each environment::
We consider there to be a single collection of Python environments on a machine,
where the collection may be different for each user of the machine. There are
three potential registry locations where the collection may be stored based on
HKEY_CURRENT_USER\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Python\<Company>\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\<Company>\<Tag>
for conflict resolution: include everything, or give priority to user
preferences.
Tools that include every installed environment, even where the Company-Tag pairs
match, should ensure users can easily identify whether the registration was
per-user or per-machine.
Tools that give priority to user preferences must ignore values from
``HKEY_LOCAL_MACHINE`` when a matching Company-Tag pair exists is in
``HKEY_CURRENT_USER``.
Official Python releases use ``PythonCore`` for Company, and the value of
``sys.winver`` for Tag. Other registered environments may use any values for
Company and Tag. Recommendations are made in the following sections.
be automatically discoverable by external tools.
Backwards Compatibility
-----------------------
Python 3.4 and earlier did not distinguish between 32-bit and 64-bit builds in
``sys.winver``. As a result, it is possible to have valid side-by-side
installations of both 32-bit and 64-bit interpreters.
To ensure backwards compatibility, applications should treat environments listed
HKEY_LOCAL_MACHINE\Software\Python\PythonCore\<Tag>
HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\<Tag>
both of the above keys, potentially resulting in three environments discovered
using the same Tag. Alternatively, a tool may determine whether the per-user
environment is 64-bit or 32-bit and give it priority over the per-machine
environment, resulting in a maximum of two discovered environments.
It is not possible to detect side-by-side installations of both 64-bit and
32-bit versions of Python prior to 3.5 when they have been installed for the
current user. Python 3.5 and later always uses different Tags for 64-bit and
32-bit versions.
Environments registered under other Company names must use distinct Tags to
support side-by-side installations. There is no backwards compatibility
allowance.
Company
-------
The Company part of the key is intended to group related environments and to
ensure that Tags are namespaced appropriately. The key name should be
alphanumeric without spaces and likely to be unique. For example, a trademarked
name, a UUID, or a hostname would be appropriate::
HKEY_CURRENT_USER\Software\Python\ExampleCorp
HKEY_CURRENT_USER\Software\Python\6C465E66-5A8C-4942-9E6A-D29159480C60
HKEY_CURRENT_USER\Software\Python\www.example.com
(``py.exe``). It does not follow this convention and should be ignored by tools.
----------
Some other registry keys are used for defining or inferring search paths under
certain conditions. A third-party installation is permitted to define these keys
under their Company-Tag key, however, the interpreter must be modified and
rebuilt in order to read these values.
Copyright
=========
This document has been placed in the public domain.
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
David Cournapeau
Mar 1, 2016, 6:38:08 AM3/1/16
to Steve Dower, python-dev
Hi Steve,
I have looked into this PEP to see what we need to do on Enthought side of things. I have a few questions:4. When a python from enthought is installed, it should never register anything in the key defined in 2.
Is this correct ?
Paul Moore
Mar 1, 2016, 8:25:08 AM3/1/16
to David Cournapeau, Steve Dower, python-dev
On 1 March 2016 at 11:37, David Cournapeau <cour...@gmail.com> wrote:
> I am not clear about 3., especially on what should be changed. I know that
> for 2.7, we need to change PC\getpathp.c for sys.path, but are there any
> other places where the registry is used by python itself ?
My understanding from the earlier discussion was that you should not
> I am not clear about 3., especially on what should be changed. I know that
> for 2.7, we need to change PC\getpathp.c for sys.path, but are there any
> other places where the registry is used by python itself ?
patch Python at all. The sys.path building via PythonPath is not
covered by the PEP and you should continue as at present. The new keys
are all for informational purposes - your installer should write to
them, and read them if looking for your installations. But the Python
interpreter itself should not know or care about your new keys.
Steve can probably clarify better than I can, but that's how I recall
it being intended to work.
Paul
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Steve Dower
Mar 1, 2016, 12:48:36 PM3/1/16
to Paul Moore, David Cournapeau, python-dev
On 01Mar2016 0524, Paul Moore wrote:
> On 1 March 2016 at 11:37, David Cournapeau <cour...@gmail.com> wrote:
>> I am not clear about 3., especially on what should be changed. I know that
>> for 2.7, we need to change PC\getpathp.c for sys.path, but are there any
>> other places where the registry is used by python itself ?
>
> My understanding from the earlier discussion was that you should not
> patch Python at all. The sys.path building via PythonPath is not
> covered by the PEP and you should continue as at present. The new keys
> are all for informational purposes - your installer should write to
> them, and read them if looking for your installations. But the Python
> interpreter itself should not know or care about your new keys.
>
> Steve can probably clarify better than I can, but that's how I recall
> it being intended to work.
> Paul
Yes, the intention was to not move sys.path building out of the
> On 1 March 2016 at 11:37, David Cournapeau <cour...@gmail.com> wrote:
>> I am not clear about 3., especially on what should be changed. I know that
>> for 2.7, we need to change PC\getpathp.c for sys.path, but are there any
>> other places where the registry is used by python itself ?
>
> My understanding from the earlier discussion was that you should not
> patch Python at all. The sys.path building via PythonPath is not
> covered by the PEP and you should continue as at present. The new keys
> are all for informational purposes - your installer should write to
> them, and read them if looking for your installations. But the Python
> interpreter itself should not know or care about your new keys.
>
> Steve can probably clarify better than I can, but that's how I recall
> it being intended to work.
> Paul
PythonCore key. It's solely about discovery by external tools.
If you want to patch your own distribution to move the paths you are
welcome to do that - there is only one string literal in getpathp.c that
needs to be updated - but it's not a requirement and I deliberately
avoided making a recommendation either way. (Though as discussed earlier
in the thread, I'm very much in favour of deprecating and removing any
use of the registry by the runtime itself in 3.6+, but still working out
the implications of that.)
Cheers,
Steve
David Cournapeau
Mar 1, 2016, 2:38:46 PM3/1/16
to Steve Dower, python-dev
On Tue, Mar 1, 2016 at 5:46 PM, Steve Dower <steve...@python.org> wrote:
Yes, the intention was to not move sys.path building out of the PythonCore key. It's solely about discovery by external tools.On 01Mar2016 0524, Paul Moore wrote:
On 1 March 2016 at 11:37, David Cournapeau <cour...@gmail.com> wrote:
I am not clear about 3., especially on what should be changed. I know that
for 2.7, we need to change PC\getpathp.c for sys.path, but are there any
other places where the registry is used by python itself ?
My understanding from the earlier discussion was that you should not
patch Python at all. The sys.path building via PythonPath is not
covered by the PEP and you should continue as at present. The new keys
are all for informational purposes - your installer should write to
them, and read them if looking for your installations. But the Python
interpreter itself should not know or care about your new keys.
Steve can probably clarify better than I can, but that's how I recall
it being intended to work.
Paul
Right. For us, continuing populating sys.path from the registry "owned" by python.org official installers is more and more untenable, because every distribution writes there, and this is especially problematic when you have both 32 bits and 64 bits distributions in the same machine.
If you want to patch your own distribution to move the paths you are welcome to do that - there is only one string literal in getpathp.c that needs to be updated - but it's not a requirement and I deliberately avoided making a recommendation either way. (Though as discussed earlier in the thread, I'm very much in favour of deprecating and removing any use of the registry by the runtime itself in 3.6+, but still working out the implications of that.)
Great, I just wanted to make sure removing it ourselves do not put us in a corner or further away from where python itself is going.
Would it make sense to indicate in the PEP that doing so is allowed (neither recommended or frowned upon) ?
David
Steve Dower
Mar 1, 2016, 3:27:58 PM3/1/16
to David Cournapeau, python-dev
On 01Mar2016 1137, David Cournapeau wrote:
> If you want to patch your own distribution to move the paths you are
> welcome to do that - there is only one string literal in getpathp.c
> that needs to be updated - but it's not a requirement and I
> deliberately avoided making a recommendation either way. (Though as
> discussed earlier in the thread, I'm very much in favour of
> deprecating and removing any use of the registry by the runtime
> itself in 3.6+, but still working out the implications of that.)
>
>
> Great, I just wanted to make sure removing it ourselves do not put us in
> a corner or further away from where python itself is going.
>
> Would it make sense to indicate in the PEP that doing so is allowed
> (neither recommended or frowned upon) ?
I was hoping not, but I suspect the question will come up again, so best
> If you want to patch your own distribution to move the paths you are
> welcome to do that - there is only one string literal in getpathp.c
> that needs to be updated - but it's not a requirement and I
> deliberately avoided making a recommendation either way. (Though as
> discussed earlier in the thread, I'm very much in favour of
> deprecating and removing any use of the registry by the runtime
> itself in 3.6+, but still working out the implications of that.)
>
>
> Great, I just wanted to make sure removing it ourselves do not put us in
> a corner or further away from where python itself is going.
>
> Would it make sense to indicate in the PEP that doing so is allowed
> (neither recommended or frowned upon) ?
to address it once in the official doc.
Michael Felt
Mar 2, 2016, 12:13:10 PM3/2/16
to pytho...@python.org
Hello all,
1) There are many lists to choose from - if this is the wrong one for
questions about packaging - please forgive me, and point me in the right
direction.
2) Normally, I have just packaged python, and then moved on. However,
recently I have been asked to help with packaging an 'easier to install'
python by people using cloud-init, and more recently people wanting to
use salt-stack (on AIX).
FYI: I have been posting about my complete failure to build 2.7.11 (
http://bugs.python.org/issue26466) - so, what I am testing is based on
2.7.10 - which built easily for me.
Going through the 'base documentation' I saw a reference to both
sys.argv and sys.path. atm, I am looking for an easy way to get the
program name (e.g., /opt/bin/python, versus ./python).
I have my reasons (basically, looking for a compiled-in library search
path to help with http://bugs.python.org/issue26439)
Looking on two platforms (AIX, my build, and debian for power) I am
surprised that sys.argv is empty in both cases, and sys.path returns
/opt/lib/python27.zip with AIX, but not with debian.
root@x064:[/data/prj/aixtools/python/python-2.7.10]/opt/bin/python
Python 2.7.10 (default, Nov 3 2015, 14:36:51) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.argv
['']
>>> sys.path
['', '/opt/lib/python27.zip', '/opt/lib/python2.7',
'/opt/lib/python2.7/plat-aix5', '/opt/lib/python2.7/lib-tk',
'/opt/lib/python2.7/lib-old', '/opt/lib/python2.7/lib-dynload',
'/opt/lib/python2.7/site-packages']
michael@ipv4:~$ python
Python 2.7.9 (default, Mar 1 2015, 13:01:00)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.argv
['']
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-powerpc-linux-gnu',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PILcompat',
'/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7']
And I guess I would be interested in getting
'/opt/lib/python2.7/dist-packages' in there as well (or learn a way to
later add it for pre-compiled packages such as cloud-init AND that those
would also look 'first' in /opt/lib/python2.7/dist-packages/cloud-init
for modules added to support cloud-init - should I so choose (mainly in
case of compatibility issues between say cloud-init and salt-stack that
have common modules BUT may have conflicts) - Hopefully never needed for
that reason, but it might also simplify packaging applications that
depend on python.
Many thanks for your time and pointers into the documentation, It is a
bit daunting :)
Michael
1) There are many lists to choose from - if this is the wrong one for
questions about packaging - please forgive me, and point me in the right
direction.
2) Normally, I have just packaged python, and then moved on. However,
recently I have been asked to help with packaging an 'easier to install'
python by people using cloud-init, and more recently people wanting to
use salt-stack (on AIX).
FYI: I have been posting about my complete failure to build 2.7.11 (
http://bugs.python.org/issue26466) - so, what I am testing is based on
2.7.10 - which built easily for me.
Going through the 'base documentation' I saw a reference to both
sys.argv and sys.path. atm, I am looking for an easy way to get the
program name (e.g., /opt/bin/python, versus ./python).
I have my reasons (basically, looking for a compiled-in library search
path to help with http://bugs.python.org/issue26439)
Looking on two platforms (AIX, my build, and debian for power) I am
surprised that sys.argv is empty in both cases, and sys.path returns
/opt/lib/python27.zip with AIX, but not with debian.
root@x064:[/data/prj/aixtools/python/python-2.7.10]/opt/bin/python
Python 2.7.10 (default, Nov 3 2015, 14:36:51) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.argv
['']
>>> sys.path
['', '/opt/lib/python27.zip', '/opt/lib/python2.7',
'/opt/lib/python2.7/plat-aix5', '/opt/lib/python2.7/lib-tk',
'/opt/lib/python2.7/lib-old', '/opt/lib/python2.7/lib-dynload',
'/opt/lib/python2.7/site-packages']
michael@ipv4:~$ python
Python 2.7.9 (default, Mar 1 2015, 13:01:00)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.argv
['']
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-powerpc-linux-gnu',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PILcompat',
'/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7']
And I guess I would be interested in getting
'/opt/lib/python2.7/dist-packages' in there as well (or learn a way to
later add it for pre-compiled packages such as cloud-init AND that those
would also look 'first' in /opt/lib/python2.7/dist-packages/cloud-init
for modules added to support cloud-init - should I so choose (mainly in
case of compatibility issues between say cloud-init and salt-stack that
have common modules BUT may have conflicts) - Hopefully never needed for
that reason, but it might also simplify packaging applications that
depend on python.
Many thanks for your time and pointers into the documentation, It is a
bit daunting :)
Michael
Brett Cannon
Mar 2, 2016, 12:21:40 PM3/2/16
to Michael Felt, pytho...@python.org
On Wed, 2 Mar 2016 at 09:12 Michael Felt <aixt...@gmail.com> wrote:
Hello all,
1) There are many lists to choose from - if this is the wrong one for
questions about packaging - please forgive me, and point me in the right
direction.
So in this instance you're after python-list since this is a general support question. But since I have an answer for you...
2) Normally, I have just packaged python, and then moved on. However,
recently I have been asked to help with packaging an 'easier to install'
python by people using cloud-init, and more recently people wanting to
use salt-stack (on AIX).
FYI: I have been posting about my complete failure to build 2.7.11 (
http://bugs.python.org/issue26466) - so, what I am testing is based on
2.7.10 - which built easily for me.
Going through the 'base documentation' I saw a reference to both
sys.argv and sys.path. atm, I am looking for an easy way to get the
program name (e.g., /opt/bin/python, versus ./python).
I have my reasons (basically, looking for a compiled-in library search
path to help with http://bugs.python.org/issue26439)
Looking on two platforms (AIX, my build, and debian for power) I am
surprised that sys.argv is empty in both cases, and sys.path returns
/opt/lib/python27.zip with AIX, but not with debian.
Did you actually build your version of Python on Debian? If not then do realize that Debian patches their version of CPython, so it wouldn't shock me if they stripped out the code that adds the zip file to sys.path. IOW don't trust the pre-installed CPython to act the same as one that is built from source.
Thomas Wouters
Mar 2, 2016, 12:47:08 PM3/2/16
to Michael Felt, Python-Dev
On Wed, Mar 2, 2016 at 3:50 AM, Michael Felt <aixt...@gmail.com> wrote:
Hello all,
1) There are many lists to choose from - if this is the wrong one for questions about packaging - please forgive me, and point me in the right direction.
It's hard to say where this belongs best, but python-list would probably have done as well.
2) Normally, I have just packaged python, and then moved on. However, recently I have been asked to help with packaging an 'easier to install' python by people using cloud-init, and more recently people wanting to use salt-stack (on AIX).
FYI: I have been posting about my complete failure to build 2.7.11 ( http://bugs.python.org/issue26466) - so, what I am testing is based on 2.7.10 - which built easily for me.
Going through the 'base documentation' I saw a reference to both sys.argv and sys.path. atm, I am looking for an easy way to get the program name (e.g., /opt/bin/python, versus ./python).
I have my reasons (basically, looking for a compiled-in library search path to help with http://bugs.python.org/issue26439)
I think the only way to get at the compiled-in search path is to recreate it based on the compiled-in prefix, which you can get through distutils. Python purposely only uses the compiled-in path as the last resort. Instead, it searches for its home relative to the executable and adds a set of directories relative to its home (if they exist).
It's not clear to me why you're focusing on these differences, as (as I describe below) they are immaterial.
Looking on two platforms (AIX, my build, and debian for power) I am surprised that sys.argv is empty in both cases, and sys.path returns /opt/lib/python27.zip with AIX, but not with debian.
When you run python interactively, sys.argv[0] will be '', yes. Since you're not launching a program, there's nothing else to set it to. 'python' (or the path to the executable) wouldn't be the right thing to set it to, because python itself isn't a Python program :)
The actual python executable is sys.executable, not sys.argv[0], but you shouldn't usually care about that, either. If you want to know where to install things, distutils is the thing to use. If you want to know where Python thinks it's installed (for debugging purposes only, really), sys.prefix will tell you.
root@x064:[/data/prj/aixtools/python/python-2.7.10]/opt/bin/python
Python 2.7.10 (default, Nov 3 2015, 14:36:51) [C] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.argv
['']
>>> sys.path
['', '/opt/lib/python27.zip', '/opt/lib/python2.7', '/opt/lib/python2.7/plat-aix5', '/opt/lib/python2.7/lib-tk', '/opt/lib/python2.7/lib-old', '/opt/lib/python2.7/lib-dynload', '/opt/lib/python2.7/site-packages']
michael@ipv4:~$ python
Python 2.7.9 (default, Mar 1 2015, 13:01:00)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.argv
['']
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-powerpc-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7']
In sys.path, you're seeing the difference between a vanilla Python and Debian's patched Python. Vanilla Python adds $prefix/lib/python27.zip to sys.path unconditionally, whereas Debian removes it when it doesn't exist. Likewise, the dist-packages directory is a local modification by Debian; in vanilla Python it's called 'site-packages' instead. The subdirectories in dist-packages that you see in the Debian case are added by .pth files installed in $prefix -- third-party packages, in other words, adding their own directories to the module search path.
And I guess I would be interested in getting '/opt/lib/python2.7/dist-packages' in there as well (or learn a way to later add it for pre-compiled packages such as cloud-init AND that those would also look 'first' in /opt/lib/python2.7/dist-packages/cloud-init for modules added to support cloud-init - should I so choose (mainly in case of compatibility issues between say cloud-init and salt-stack that have common modules BUT may have conflicts) - Hopefully never needed for that reason, but it might also simplify packaging applications that depend on python.
A vanilla Python (or non-Debian-built python, even) has no business looking in dist-packages. It should just use site-packages. Third-party packages shouldn't care whether they're installed in site-packages or dist-packages, and instead should use distutils one way or another (if not by having an actual setup.py that uses distutils or setuptools, then at least by querying distutils for the installation directory the way python-config does).
Many thanks for your time and pointers into the documentation, It is a bit daunting :)
Michael
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Thomas Wouters <tho...@python.org>
Hi! I'm an email virus! Think twice before sending your email to help me spread!
Hi! I'm an email virus! Think twice before sending your email to help me spread!
Michael Felt
Mar 13, 2016, 12:04:53 PM3/13/16
to Thomas Wouters, Python-Dev
My question is to help me understand where to look for default libraries
in order to work on a patch.
That has come a long way, but it is stuck now with/by something else I
need to learn to debug (to find where the non-zero exit status comes
from during a build).
As a packager I hope to be as 'vanilla' as possible from the python
perspective.
Another time (another list!) I shall ask about what goes into python.zip
Michael
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
https://mail.python.org/mailman/listinfo/python-dev
Reply all
Reply to author
Forward
0 new messages