[Python-Dev] Have virtual environments led to neglect of the actual environment?

[Random832]

I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:

Christian Heimes wrote:
> Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.

I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?

I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.

How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/KMRNKSVEPHAXA7BHFT4PWX4EKWYUF4G7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Paul Bryan]

I think it's a classic case of dependency hell.

OS packagers are rebundling Python packages as OS packages and expressing their own OS-package dependency graphs. Then, you sudo pip install something that has a conflicting dependency, it bypasses OS packaging, and *boom*.

I find tools like pipx go a long way to solve this, as they install a Python package and all of its dependencies in its own venv. This is great for Python apps, and (kinda) treats them like apps on platforms like Android, where all app dependencies are bundled and isolated from others.

I think it would great if OS vendors did something similar to pipx for Python-based apps: bundle the app and all of its dependencies into its own venv.

 

On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:

I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:

Christian Heimes wrote:

Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.

I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?

I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.

How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?

_______________________________________________

Python-ideas mailing list -- python...@python.org

To unsubscribe send an email to python-id...@python.org

https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/KMRNKSVEPHAXA7BHFT4PWX4EKWYUF4G7/

[Jonathan Goble]

On Tue, Feb 23, 2021 at 7:48 PM Random832 <rand...@fastmail.com> wrote:
>
> I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
>
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
>

> [snip]

>
> How is it that virtual environments have become so indispensable, that no-one considers installing libraries centrally to be a viable model anymore? Are library maintainers making breaking changes too frequently, reasoning that if someone needs the old version they can just venv it? Is there some other cause?

I can't speak for distributors or maintainers [1], but I can speak for
myself as a user. I run Debian testing (currently bullseye as that is
preparing for release) as my daily OS on my personal laptop, used for
personal matters and school assignments (I'm a university computer
science student in my senior year).

I don't use the system Python for anything of my own, whether it's a
school assignment or a personal project, precisely because I don't
want to risk screwing something up. Rather, I maintain a clone/fork of
the official CPython GitHub repo, and periodically build from source
and `make altinstall` into `~/.local/`. The `python3` command
continues to refer to the system Python, while `python3.8`, etc. refer
to the ones installed in my home folder. To the latter I make symlinks
for `py38`, `py39`, etc., and just `py` (and `pip`) for the one I use
most often (usually the latest stable release). I typically have
multiple versions installed at once since different scripts/projects
run on different versions at different times. Given this setup, I can
just do a simple `pip install spam` command and I don't need either
`sudo` or `--user`, nor do I need virtual envs.

While the average person would probably not clone the GitHub repo and
build that way, it's not terribly unreasonable for an inclined person
to do the same with a tarball downloaded from python.org, and so I
doubt I'm the only one with this type of setup.

Just some food for thought.

[1] Technically I am a library maintainer since I have a couple
projects on PyPI, but those are mostly unused and more or less
abandoned at this point, and neither ever reached the point where I
could consider graduating them from beta status. Most of what I work
with these days is private personal code or school assignments.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/M6OGQXKLG27Y6KA5L5UFBWA2NAZNBOHL/

[Emily Bowman]

On Tue, Feb 23, 2021 at 4:51 PM Random832 <rand...@fastmail.com> wrote:

Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?

The alternative to virtualenv is the solution most software takes for, eg, Java: Package a complete binary distribution internally to eliminate any dependency on the system package -- and allow the software to pollute their own package in any way they want without affecting the rest of the system, while potentially becoming very out of date with security patches. Virtualenv allows software to lockstep and pollute their own runtime of Python almost all they want while still sharing the actual base install that gets regular security and bugfix updates. (Of course, even the hint of possibility that anything about the runtime could change and cause downtime is too much for many enterprise Java systems, no matter how severe the security update, so even ported to Python they wouldn't settle for that, they'd still package the entire interpreter internally.)

Since telling application developers "no, don't do that, just get along" doesn't work, no matter how loudly you say it, virtualenv is a happy middle ground. A few core OS maintainers are much more likely to listen, so the yelling about splitting up or overly customizing the stdlib achieves results.

-Em

[Wes Turner]

First, pip+venv is not sufficient for secure software deployment:
something must set appropriate permissions so that the application
cannot overwrite itself and other core libraries (in order to
eliminate W^X violations (which e.g. Android is solving by requiring
all installed binaries to come from an APK otherwise they won't and
can't be labeled with the SELinux extended file atrributes necessary
for a binary to execute; but we don't have binaries, we have an
interpreter and arbitrary hopefully-signed somewhere source code, at
least)).

Believe it or not, this is wrong:

```bash
# python -m venv httpbin || virtualenv httpbin
# source httpbin/bin/activate
mkvirtualenv httpbin

pip install httpbin gunicorn
gunicorn -b 0.0.0.0:8001 httpbin:app

# python -m webbrowser http://127.0.0.1:8001
```

It's wrong - it's insecure - because the user executing the Python
interpreter (through gunicorn, in this case) can overwrite the app.
W^X: has both write and execute permissions. What would be better?

This would be better because pip isn't running setup.py as root (with
non-wheels) and httpbin_exec can't modify the app interpreter or the
code it loads at runtime:

```bash
useradd httpbin # also creates a group also named 'httpbin'
sudo -u httpbin sh -c ' \
python -m venv httpbin; \
umask 0022; \
./httpbin/bin/python -m pip install httpbin gunicorn'

useradd httpbin_exec -G httpbin
sudo -u httpbin_exec './httpbin/bin/gunicorn -b 0.0.0.0:8001 httpbin:app'
```

This would be better if it worked, though there are a few caveats:

```bash
sudo apt-get install python-gunicorn python-httpbin
sudo -u nobody /usr/bin/gunicorn -b 0.0.0.0:8001 httpbin:app
```

1. Development is impossible:
- You can't edit the code in /usr/lib/python3.n/site-package/ without
root permissions.
- You should not be running an editor as root.
- You can edit distro-package files individually with e.g. sudoedit
(and then the GPG-signed package file checksums will fail when you run
`debsums` or `rpm -Va` because you've edited the file and that's
changed the hash).

- Non-root users cannot install python packages without having someone
repack (and sign it) for them.

- What do I need to do in order to patch the distro's signed repack of
the Python package released to PyPI?
- I like how Fedora pkgs and conda-forge have per-package git repos now.
- Conda-forge has a bot that watches PyPI for new releases and tries
sending an automated PR.
- If I send a PR to the main branch of the source repo and it gets
merged, how long will it be before there's a distro repack built and
uploaded to the distro package index?

2. It should be installed in a chroot/jail/zone/container/context/vm
so that it cannot read other data on the machine.
The httpbin app does not need read access to /etc/shadow, for example.
Distro package installs are not - either - sandboxed.

To pick on httpbin a bit more, the httpbin docs specify that httpbin
should be run as a docker container:

```bash
docker run -p 80:8001 kennethreitz/httpbin
```

Is that good enough? We don't know, we haven't reviewed:

- the Dockerfile
- it says `FROM ubuntu:18.04`, which is fortunately an LTS release.
But if it hasn't been updated this month, it probably has the sudo bug
that enabled escalation to root (which - even in a container - is bad
because it could obnoxiously just overwrite libc, for example, and
unless the container is rebuilt or something runs `debsums`, nothing
will detect that data integrity error)
- the requirements.txt / setup.py:install_requires / Pipfile[.lock] dependencies
- does it depend upon outdated pinned exact versions?
- Is there an SBOM (Software Bill of Materials) that we can review
against known vulnerability databases?

How do I know that:

- The packages I have installed are not outdated and unpatched against
known vulnerabilities
- The files on disk are exactly what should be in the package
- The app_exec user can't overwrite the binary interpreter or the
source files it loads at runtime
- There won't be unreviewed code running as root (including at install time)
- All Python package dependencies are available as wheels (that
basically only need to be unzipped)
- The ensemble of dependencies which I've miraculously assembled is
available on the target platform(s)
- The integration tests for my app pass with each combination of
dependencies which satisfy the specified dependency constraints
- I can edit things and quickly re-test
- Each dependency is signed by a key that's valid for that dependency

So, if pip is insufficient for secure software deployment, what are
pro teams using to build signed, deployable artifacts with fresh,
upgaded dependencies either bundled in or loosely-referenced?

- Bazel (from Google's internal Blaze) builds from BUILD files.
- https://github.com/dropbox/dbx_build_tools
- Pantsbuild, Buck
- zipapps
- FPM can apparently package up an entire virtualenv; though IDK how
good it is at permissions?
https://github.com/jordansissel/fpm/blob/master/lib/fpm/package/virtualenv.rb

As an open source maintainer, there are very many potential
environments to release builds for.
Manylinux docker images (and auditwheel, delocate, and *cibuildwheel*)
are a response to extreme and somewhat-avoidable complexity.
https://github.com/joerick/cibuildwheel

Distro packagers can and do build upon e.g. pip; which is great for
development but not sufficient for production deployment due to lack
of support for file permissions, extended file attributes, checksums,
cryptographic signatures, and due to running setup.py as the install
user for non-wheel packages.

There are many deployment stories now: pull/push, configuration
management systems, venvs within containers within VMs. For your
favorite distro, how do I get from cibuildwheel to a signed release
artifact in your package index; and which keys can sign for what?

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OF3ZZUKEZC5D3JTK3VQX37PV75ZBQWZX/

[Wes Turner]

FWIW, Distro repacks are advantageous in comparison to
"statically-bundled" releases that for example bundle in an outdated
version of OpenSSL, because when you `apt-get upgrade -y` that should
upgrade the OpenSSL that all the other distro packages depend upon.

Here's something that doesn't get called as frequently as `apt-get upgrade -y`:

```bash
pip install -U certifi
```

https://github.com/certifi/python-certifi (Mozilla's CA bundle
extracted into a Python package)

```bash
apt-get install -y ca-certificates
dnf install -y ca-certificates
```

--
Wes Turner
https://westurner.org
https://wrdrd.com/docs/consulting/knowledge-engineering

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/YE64OOO6STWXUDYOHRAYK4A3BSEYQTF3/

[Michał Górny]

On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:

> I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
>
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
>
> I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
>
> I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.

The problem is a little deeper and the Debian solution doesn't really
solve all of it.

Yes, pip installing into the same directory as the package manager
is a problem. It's a problem to the point that I'm patching pip
in Gentoo to explicitly block that. We've gotten far too many bug
reports about people's systems suddenly being horribly broken after they
used pip.

While using two directories can prevent pip from directly overwriting
system packages, you still can't expect two independent package managers
to work simultaneously unless they can communicate with each other to
prevent conflicts. If pip installs a different version of the same
package as the package manager, which one is supposed to be used?
Whichever choice you make, you'll bound to eventually break dependency
graph of some package.

--
Best regards,
Michał Górny

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/QRARBX6CJBV6ZVDFVOVLHKZJ5P44NPTC/

[Antoine Pitrou]

On Tue, 23 Feb 2021 20:29:52 -0500
Jonathan Goble <jcgo...@gmail.com> wrote:
>
> I can't speak for distributors or maintainers [1], but I can speak for
> myself as a user. I run Debian testing (currently bullseye as that is
> preparing for release) as my daily OS on my personal laptop, used for
> personal matters and school assignments (I'm a university computer
> science student in my senior year).
>
> I don't use the system Python for anything of my own, whether it's a
> school assignment or a personal project, precisely because I don't
> want to risk screwing something up. Rather, I maintain a clone/fork of
> the official CPython GitHub repo, and periodically build from source
> and `make altinstall` into `~/.local/`.

For the record, instead of building Python by hand, you could use a
distribution such as Anaconda or the community-maintained conda-forge.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/3TXDWMNBTQLCSEEG5XKMPJHJJQ4BRWWF/

[Henk-Jaap Wagenaar]

On Wed, 24 Feb 2021 at 10:18, Antoine Pitrou <ant...@python.org> wrote:

On Tue, 23 Feb 2021 20:29:52 -0500
Jonathan Goble <jcgo...@gmail.com> wrote:
>
> I can't speak for distributors or maintainers [1], but I can speak for
> myself as a user. I run Debian testing (currently bullseye as that is
> preparing for release) as my daily OS on my personal laptop, used for
> personal matters and school assignments (I'm a university computer
> science student in my senior year).
>
> I don't use the system Python for anything of my own, whether it's a
> school assignment or a personal project, precisely because I don't
> want to risk screwing something up. Rather, I maintain a clone/fork of
> the official CPython GitHub repo, and periodically build from source
> and `make altinstall` into `~/.local/`.

For the record, instead of building Python by hand, you could use a
distribution such as Anaconda or the community-maintained conda-forge.

Regards

Antoine.

I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been quite happy with that, and it basically does what Jonathan does manually: clone the github repo and build python from scratch.

[Christian Heimes]

On 24/02/2021 08.03, Michał Górny wrote:
> On Tue, 2021-02-23 at 19:45 -0500, Random832 wrote:
>> I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
>>
>> Christian Heimes wrote:
>>> Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
>>
>> I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?
>>
>> I think that installing packages into the actual python installation, both via distribution packaging tools and pip [and using both simultaneously - the Debian model of separated dist-packages and site-packages folders seems like a reasonable solution to this problem] can and should be a supported paradigm, and that virtual environments [or more extreme measures such as shipping an entire python installation as part of an application's deployment] should ideally be reserved for the rare corner cases where that doesn't work for some reason.
>
> The problem is a little deeper and the Debian solution doesn't really
> solve all of it.
>
> Yes, pip installing into the same directory as the package manager
> is a problem. It's a problem to the point that I'm patching pip
> in Gentoo to explicitly block that. We've gotten far too many bug
> reports about people's systems suddenly being horribly broken after they
> used pip.
>
> While using two directories can prevent pip from directly overwriting
> system packages, you still can't expect two independent package managers
> to work simultaneously unless they can communicate with each other to
> prevent conflicts. If pip installs a different version of the same
> package as the package manager, which one is supposed to be used?
> Whichever choice you make, you'll bound to eventually break dependency
> graph of some package.

Thanks Michał, you have explained the situation well.

Most Linux distros have two separate directories for distro packages and
"sudo pip install" packages. This is a great thing!

Separate directories don't prevent clashes and system breakage. But they
provide an easy way to *recover* from a broken system. The package
manager of several Linux distributions is written in Python. If your
system Python is broken, your package manager is likely busted as well.
The split directories allow ysers to fix their system with a single
command like "sudo rm -rf /usr/local/lib/python*" (paths may vary).

The issue is: Split distro/pip directories are not baked into Python or
pip. Distros have their own set of downstream patches that modify the
behavior of Python. Some of the patches have caused issues in the past.

I would like to see this feature implemented in Python's core, so distro
packages have to do less patching and upstream pip can test against it.

Christian

PS: I wrote PEP 370 for "install --user" 13 years ago before we had
virtual envs.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/ZYXP5FARQIYQQTW46UFWVAXQRTFNFRFT/

[Christian Heimes]

On 24/02/2021 11.52, Stéfane Fermigier wrote:
> I love pipx and I'm glad it exists at this point because it make 
>
> The main issue is that each virtualenv takes space, lots of space.
>
> I have currently 57 apps installed via pipx on my laptop, and the 57
> environments take almost 1 GB already.
>
>  ~  cd .local/pipx/venvs/
>  ~/.l/p/venvs  ls
> abilian-tools/  concentration/  gitlabber/      pygount/        sphinx/
> ansible/        cookiecutter/   httpie/         pyinfra/        tentakel/
> assertize/      cruft/          isort/          pylint/         tlv/
> autoflake/      cython/         jupyterlab/     pyre-check/     towncrier/
> black/          dephell/        lektor/         pytype/         tox/
> borgbackup/     docformatter/   md2pdf/         pyupgrade/      twine/
> borgmatic/      flake8/         medikit/        radon/          virtualenv/
> bpytop/         flit/           mypy/           re-ver/         virtualfish/
> check-manifest/ flynt/          nox/            sailboat/       vulture/
> clone-github/   gh-clone/       pdoc3/          salvo/
> cloneall/       ghtop/          pdocs/          shed/
> com2ann/        gitchangelog/   pybetter/       sixer/
>  ~/.l/p/venvs  du -sh .
> 990M.
>  ~/.l/p/venvs  ls | wc
>       57      57     475
>
> There is probably a clever way to reuse common packages (probably via
> clever symlinking) and reduce the footprint of these installations. 

There are tools like https://rdfind.pauldreik.se/rdfind.1.html that
create hard links to deduplicate files. Some files systems have
deduplicated baked in, too.

Christian

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/CO6GV2CRDKBJMLE7DZVVQ4AMIPSKPMCJ/

[Paul Moore]

On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier <s...@fermigier.com> wrote:
> There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations.

Ultimately the problem is that a general tool can't deal with
conflicts (except by raising an error). If application A depends on
lib==1.0 and application B depends on lib==2.0, you simply can't have
a (consistent) environment that supports both A and B. But that's the
rare case - 99% of the time, there are no conflicts. One env per app
is a safe, but heavy handed, approach. Managing environments manually
isn't exactly *hard*, but it's annoying manual work that pipx does an
excellent job of automating, so it's a disk space vs admin time
trade-off.

As far as I know, no-one has tried to work on the more complex option
of sharing things (pipx shares the copies of pip, setuptools and wheel
that are needed to support pipx itself, but doesn't extend that to
application dependencies). It would be a reasonable request for pipx
to look at, or for a new tool, but I suspect the cost of implementing
it simply outweighs the benefit ("disk space is cheap" :-))

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4EAGDI66MG4WPNCHCOVHT4VHBWQUNRNP/

[Paul Moore]

On Wed, 24 Feb 2021 at 13:12, Antoine Pitrou <ant...@python.org> wrote:
>
> On Wed, 24 Feb 2021 13:47:40 +0100
> Stéfane Fermigier <s...@fermigier.com> wrote:
> > The 3rd solution is probably the best of the 3, but the sharing mechanism
> > still needs to be specified (and, if needed, implemented) properly.
>
> I wouldn't want to repeat myself too often, but conda and conda-based
> distributions already have sharing through hardlinks (or, on Windows,
> whatever is available) baked-in, assuming you install your software
> from conda packages.
>
> That also applies to non-Python packages, and to python itself (which
> is just a package like any other).

I'm not sure conda solves the problem of *application* distribution,
though, so I think it's addressing a different problem. Specifically,
I don't think conda addresses the use case pipx is designed for.

Although to be fair, this conversation has drifted *way* off the
original topic. Going back to that, my view is that Python does not
have a good solution to the "write your application in Python, and
then distribute it" scenario. Shipping just the app to be run on an
independently installed runtime results in the conflicting
dependencies issue. Shipping the app with bundled dependencies is
clumsy, mostly because no-one has developed tools to make it easier.
It also misses opportunities for sharing libraries (reduced
maintenance, less disk usage...). Shipping the app with a bundled
interpreter and libraries is safest, but hard to do and even more
expensive than the "bundled libraries" approach.

I'd love to see better tools for this, but the community preferred
approach seems to be "ship your app as a PyPI package with a console
entry point" and that's the approach pipx supports.

I don't use Linux much, and I'm definitely not familiar with Linux
distribution tools, but from what I can gather Linux distributions
have made the choices:

1. Write key operating system utilities in Python.
2. Share the Python interpreter and libraries.
3. Expose that Python interpreter as the *user's* default Python.

IMO, the mistake is (3) - because the user wants to install Python
packages, and not all packages are bundled by the distribution (or if
they are, they aren't updated quickly enough for the user), users want
to be able to install packages using Python tools. That risks
introducing unexpected library versions and/or conflicts, which breaks
the OS utilities, which expect their requirements to be respected
(that's what the OS packaging tools do).

Hindsight is way too easy here, but if distros had a "system Python"
package that OS tools depend on, and which is reserved for *only* OS
tools, and a "user Python" package that users could write their code
against, we'd probably have had far fewer issues (and much less FUD
about the "using sudo pip breaks your OS" advice). But it's likely way
too late to argue for such a sweeping change.

*Shrug* I'm not the person to ask here. My view is that I avoid using
Python on Linux, because it's *way* too hard. I find it so much easier
to work on Windows, where I can install Python easily for myself, and
I don't have to fight with system package managers, or
distribution-patched tools that don't work the way I expect. And
honestly, on Windows, there's no "neglect of the system environment"
to worry about - if you want to install Python, and use pip to install
packages into that environment for shared use, it works fine. People
(including me) use virtual environments for *convenience* on Windows,
not because it's a requirement.

Paul
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/J7U6525AZAW4P5ZYH5WLK5IDF6TCH73O/

[Random832]

On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
> Separate directories don't prevent clashes and system breakage. But they
> provide an easy way to *recover* from a broken system.

I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.

Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installation.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/K3DB2MSBWPDEK3O4UZ5VEQVKW7BFZDRU/

[Random832]

On Wed, Feb 24, 2021, at 09:08, Paul Moore wrote:
> I don't use Linux much, and I'm definitely not familiar with Linux
> distribution tools, but from what I can gather Linux distributions
> have made the choices:
>
> 1. Write key operating system utilities in Python.
> 2. Share the Python interpreter and libraries.
> 3. Expose that Python interpreter as the *user's* default Python.

I think 1 *partially* mischaracterizes the problem, because any "system python" would logically be used by *every application written in python [or that embeds python] distributed by the OS's package management*, not just by "key operating system utilities". To suggest otherwise implies that they should not distribute any python applications at all.

That also makes asking all of their package maintainers to change their #! line to point at a different interpreter [or to pass an option, as I had suggested in another post] a more significant ask than the "just change a few core utilities" that some people seem to be assuming it would be. It also means that making a "system python" does not remove the need to distribute the largish subset of python *libraries* that they distribute, because even when these libraries aren't used by key OS utilities, they are still used by packaged applications.

[this, in turn, means we don't want the user's default python environment to stand entirely separate from the system python, or we'll start getting complaints along the lines of "I apt-get installed numpy, why can't I import it in my python interpreter?" - particularly from users who don't really care if it runs a couple versions behind]

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/F7YNRI7LWU2VLS6AU6PQZUHHA3M5WFQJ/

[Christian Heimes]

On 24/02/2021 15.16, Random832 wrote:
> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
>> Separate directories don't prevent clashes and system breakage. But they
>> provide an easy way to *recover* from a broken system.
>
> I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.
>
> Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installation.

Yes, you are describing one major aspect of my idea for a system Python
interpreter. I'm happy to read that other users are coming to similar
conclusions. Instead of an option I'd create a new executable to lock
down additional things (e.g. isolated mode, code verification hook). A
separate executable would also allow distros to provide a stripped down
interpreter that does not cause bad user experience.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/F5NCGF7EWYDMBAHPNDFNWWKSE2GRKHLA/

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 1:47 PM Stéfane Fermigier <s...@fermigier.com> wrote:

So IMHO the best way to implement solution 3 would be by using some variant of the approach popularized by Nix (repository of immutable packages + links to each virtualenv).

Another benefit of this kind of approach, besides sparing disk space, could be similar improvements in terms of installation time (and even bigger improvements when reinstalling a package, which happens all the time when developing). 

  S.

--

Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/

Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/

Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/

[Stéfane Fermigier]

I love pipx and I'm glad it exists at this point because it make 

The main issue is that each virtualenv takes space, lots of space.

I have currently 57 apps installed via pipx on my laptop, and the 57 environments take almost 1 GB already.

 ~  cd .local/pipx/venvs/

 ~/.l/p/venvs  ls

abilian-tools/  concentration/  gitlabber/      pygount/        sphinx/

ansible/        cookiecutter/   httpie/         pyinfra/        tentakel/

assertize/      cruft/          isort/          pylint/         tlv/

autoflake/      cython/         jupyterlab/     pyre-check/     towncrier/

black/          dephell/        lektor/         pytype/         tox/

borgbackup/     docformatter/   md2pdf/         pyupgrade/      twine/

borgmatic/      flake8/         medikit/        radon/          virtualenv/

bpytop/         flit/           mypy/           re-ver/         virtualfish/

check-manifest/ flynt/          nox/            sailboat/       vulture/

clone-github/   gh-clone/       pdoc3/          salvo/

cloneall/       ghtop/          pdocs/          shed/

com2ann/        gitchangelog/   pybetter/       sixer/

 ~/.l/p/venvs  du -sh .

990M .

 ~/.l/p/venvs  ls | wc

      57      57     475

There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations. 

Still, I'm glad that pipx exists as it is now, and that it has been packaged on Ubuntu 20.04 and later (and probably other distros as well).

Having pipx (or something similar) installed by the distro, and the distro focussed on packaging only the packages that are needed for its own sake, means that we could go past the controversies between the Python community and the Debian (or other distros) packagers community, which are based on different goals and assumption, such as this one: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e

  S.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/2VYJAJHHGOOPFQA6U7ATBKREMAC66CXW/

Code of Conduct: http://python.org/psf/codeofconduct/

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 11:43 AM Henk-Jaap Wagenaar <wagenaar...@gmail.com> wrote:

I've been using pyenv (on MacBooks to be fair, not Linux/Debian) and been quite happy with that, and it basically does what Jonathan does manually: clone the github repo and build python from scratch.

Also using pyenv myself, on platforms where either, or both, new and old version of Python are not available (so on Ubuntu I'm sticking with Anthony Sottile's deadsnakes.)

But I'm not using it to manage virtual envs. For my own development projects, I'm using VirtualFish since I've switched to Fish as my main shell, but I was happy with VirtualenvWrapper when I was still using bash or zsh. I can't live without VirtualFish's autoactivation plugin (which activate a given virtualenv each time you enter your project). There was something similar for Zsh though I don't remember which. Sometimes I forget that I'm not in the right folder, or that I didn't create the virtualenv, or that I removed it in the case of an old project, and I end up polluting my default Python install after a 'pip install ...'. No big deal but always a bit unpleasant.

And as I wrote before, I'm now using pipx to install the apps that I use.

This works not so bad (except for some quirks now and then) but it took quite some time to select the right (for me) tools, to set up the proper config files on my computer, and it can be annoying when working on a server which has not been set up the same way. It also takes some time to explain to collaborators, especially interns or juniors.

So obviously there is room for improvement, especially in terms of simplicity for beginners.

  S.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OZRHP3Q6EZNCPCTEEXLZGOCTGY6B7YFW/

Code of Conduct: http://python.org/psf/codeofconduct/

[Stéfane Fermigier]

On Wed, Feb 24, 2021 at 12:42 PM Paul Moore <p.f....@gmail.com> wrote:

On Wed, 24 Feb 2021 at 10:55, Stéfane Fermigier <s...@fermigier.com> wrote:
> There is probably a clever way to reuse common packages (probably via clever symlinking) and reduce the footprint of these installations.

Ultimately the problem is that a general tool can't deal with
conflicts (except by raising an error). If application A depends on
lib==1.0 and application B depends on lib==2.0, you simply can't have
a (consistent) environment that supports both A and B. But that's the
rare case - 99% of the time, there are no conflicts. One env per app
is a safe, but heavy handed, approach. Managing environments manually
isn't exactly *hard*, but it's annoying manual work that pipx does an
excellent job of automating, so it's a disk space vs admin time
trade-off.

There are three ways to approach the question:

1) Fully isolated envs. The safest option but uses the most space.

2) Try to minimise the number of dependencies installed by interpreting the requirements specification in the looser way possible. This is both algorithmically hard (see https://hal.archives-ouvertes.fr/hal-00149566/document for instance, or the more recent https://hal.archives-ouvertes.fr/hal-03005932/document ) and risky, as you've noted.

3) But the best way IMHO is to compute dependencies for each virtualenv independently from the others, but still share the packages, using some indirection mechanisms (hard links, symlinks or some Python-specific constructs) when the versions match exactly.

The 3rd solution is probably the best of the 3, but the sharing mechanism still needs to be specified (and, if needed, implemented) properly.

I've tried Christian's suggestions of using rdfind on my pipx installation, and it claims to reduce the footprint by 30% (nice, but less than I expected. This would however scale better with the number of installed packages).

I'm not sure this would be practical in reality, OTOH, because I think there is a serious risk of breakage each time I would upgrade one of the packages (via 'pipx upgrade-all' for instance).

So IMHO the best way to implement solution 3 would be by using some variant of the approach popularized by Nix (repository of immutable packages + links to each virtualenv).

  S.

[Steve Dower]

On 2/24/2021 4:26 PM, Christian Heimes wrote:
> On 24/02/2021 15.16, Random832 wrote:
>> On Wed, Feb 24, 2021, at 06:27, Christian Heimes wrote:
>>> Separate directories don't prevent clashes and system breakage. But they
>>> provide an easy way to *recover* from a broken system.
>>
>> I think it could be turned into a way to prevent them by A) having site-packages always take precedence over dist-packages [i believe this is already the case] in normal usage and B) providing an option to the interpreter, used by system scripts, to exclude site-packages entirely from the path.
>>
>> Basically, site-packages would effectively be layered on top of "Lib + dist-packages" in a similar way to how a venv is layered on top of the main python installation - the inverse of the suggestion someone else in the thread made for the system python to be a venv. This wouldn't *exactly* be a venv because it wouldn't imply the other things that entering a venv does such as "python" [and script names such as pip] being an alias for the correct version of python, but it would provide the same kind of one-way isolation, whereby the "system environment" can influence the "normal environment" and not vice-versa, in the same way that packages installed in the main installation affect a venv [unless system-site-packages is disabled] but the venv obviously has no effect on the main installation.
>
> Yes, you are describing one major aspect of my idea for a system Python
> interpreter. I'm happy to read that other users are coming to similar
> conclusions. Instead of an option I'd create a new executable to lock
> down additional things (e.g. isolated mode, code verification hook). A
> separate executable would also allow distros to provide a stripped down
> interpreter that does not cause bad user experience.

I mean, this is _precisely_ what PEP 370 defines (including the "-s"
option and PYTHONNOUSERSITE env variable to provide that one-way isolation).

Is the problem that pip doesn't use it by default? Or that the distros
went and made patches for the runtime rather than patching pip? (For
Windows installs from the Store, where even admin rights can't do an
all-users package install, I added a new config file location for pip to
change this default, but a patch would also have worked.)

Maybe we need an easier way to patch the location of user site packages?
I also had to do this for the Store install on Windows, and it's a
little bit of a hack... but maybe having an official recommendation
would help encourage distributors to use the mechanism?

Cheers,
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/IHEECOXYDPJ6ZQJE2QTGPDOOCOP7J37A/

[Paul Moore]

On Wed, 24 Feb 2021 at 18:25, Steve Dower <steve...@python.org> wrote:
> I mean, this is _precisely_ what PEP 370 defines (including the "-s"
> option and PYTHONNOUSERSITE env variable to provide that one-way isolation).
>
> Is the problem that pip doesn't use it by default?

Note that these days, pip *does* use usersite by default if
site-packages isn't writable. Of course, if users run pip under sudo,
that invalidates that mechanism :-(

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/UE7W7IFPNAPTU64TNTHXZ5GUCLOSMUJV/

[Christian Heimes]

(sorry for terse reply, I'm heading out.)

Linux distros want an additional layer so the can differentiate between
Python packages installed by their package manager and Python packages
installed with "sudo pip install".

Python has two default site-package directories (from highest to lowest
precedence):

1. ~/.local/lib/python3.9/site-packages
2. /usr/lib/python3.9/site-packages

(1) is defined by site.getusersitepackages(), (2) site.getsitepackages()

Linux distro have additional site-package directories. My Fedora system
X86_64 system with multiarch has three additional candidates.

1. ~/.local/lib/python3.9/site-packages
2. /usr/local/lib64/python3.9/site-packages
3. /usr/local/lib/python3.9/site-packages
4. /usr/lib64/python3.9/site-packages
5. /usr/lib/python3.9/site-packages

The "lib" directories are for pure Python packages whiel the "lib64"
directories contain X86_64 native code extensions (aka shared
libraries). The "/usr/lib*" directories are used for distro packages.
The downstream patch [1] ensures that "sudo pip install" installs
packages into "/usr/local/lib". AFAIK "/usr/local/lib64" is not used.

Debian has a similar mechanism to provide
"/usr/lib/python3/dist-packages" [2].

A hypothetical /usr/bin/system-python3 interpreter would only use (4)
and (5) as site-package directories. A user-facing /usr/bin/python3
would use (1) to (5).

Christian

[1]
https://src.fedoraproject.org/rpms/python3.9/blob/rawhide/f/00251-change-user-install-location.patch
[2]
https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RZACFNIDET7FZJG6LC25EAPWCR7ADFUE/

[Christian Heimes]

PS: Fedora patch [1] uses -s / PYTHONNOUSERSITE to exclude
"/usr/local/lib*" site-package directories.

$ python3 -c 'import site; print(site.getsitepackages())'
['/usr/local/lib64/python3.9/site-packages',
'/usr/local/lib/python3.9/site-packages',
'/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']

$ python3 -s -c 'import site; print(site.getsitepackages())'
['/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/EKYM66AZYDSRLTD5O4UOGC7FXKBB2VVL/

[Wes Turner]

Would it be better to have an (os, os_release, arch) to [site_packages_dirs] map within CPython or are the patches too varied to be replaced by a centralized map and a runtime conditional? 

For development convenience, having writeable site-packages dirs before unwriteable site-packages is easy;

But for security and production deployment, site-packages maybe shouldn't be writeable at all by default because that's doing it wrong?

[Mike Miller]

On 2021-02-24 02:52, Stéfane Fermigier wrote:
> I have currently 57 apps installed via pipx on my laptop, and the 57
> environments take almost 1 GB already.

I never understood the fear around version conflicts. Perhaps it has to do with
the decline of sys-admin skills over the years? So, the strategy above feels
like swatting a fly with a sledgehammer to me. Same as with a venv for every
app. :-)

Rather than installing every package this way, why not wait until a conflict
actually occurs?

Personally, I rarely have such conflicts, maybe every few years or so. When it
happens, I fix them by uninstalling the offender and putting the more difficult
one into the venv or pipx. Right now I only have one, a giant app from work
that uses pipenv, and it's fine.


Now what about sudo and all that? Well, I used it in the old days because
that's what the instructions said. But, to be honest, it never made any sense.
I haven't shared a computer in decades, and when we did we used NFS for common
tools, so it never saved any disk space.

Pip (and easy_install?) dragged their feet for years to properly support user
installs (should have been default) but once they did I didn't look back. I
dump almost all packages to user, which gets cleaned up every other year when
the distro moves to the next version. The strategy has been working well for a
long time.

So, --user works at the low end, and containers for the high end. Honestly, I
don't have much of a use case any longer for venvs.

-Mike

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/Z2WSO5LC7WRX24XT5WERYHSA2UONQKB6/

[Peter Wang]

On Wed, Feb 24, 2021 at 8:50 PM Mike Miller <pytho...@mgmiller.net> wrote:

I never understood the fear around version conflicts. 

With binary extension modules, version conflicts lead to (at best) runtime segfault and (at worst) subtle *data* bugs that return incorrect results.  There are also deeper concerns around security and reproducibility.

 

Perhaps it has to do with
the decline of sys-admin skills over the years?

Many millions of users of new Python users show up every year, using the language and its powerful ecosystem for data analytics and scientific computing, and they have no hope of having sys-admin skills.

-Peter

[Wes Turner]

The challenge with version conflicts is often less that you need to go update the constraints (which has little to do with sysadmin'ing, TBH) and more that you have insufficient *integration tests* and you're relying upon something else running the per-package tears.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/TYZ6QT4BVQY7SVXV6E63YKNV6SCNFZ7V/

[Fred Drake]

On Thu, Feb 25, 2021 at 5:35 AM Wes Turner <wes.t...@gmail.com> wrote:

The challenge with version conflicts is often less that you need to go update the constraints (which has little to do with sysadmin'ing, TBH) and more that you have insufficient *integration tests* and you're relying upon something else running the per-package tears.

Sometimes, auto-correct really does understand!

Your point is right on target, and really can't be understated. 

  -Fred

--

Fred L. Drake, Jr.    <fred at fdrake.net>
"There is nothing more uncommon than common sense."
  --Frank Lloyd Wright

[Mike Miller]

On 2021-02-24 21:08, Peter Wang wrote:
> With binary extension modules, version conflicts lead to (at best) runtime
> segfault and (at worst) subtle *data* bugs that return incorrect results.  There
> are also deeper concerns around security and reproducibility.

If your app has requirements, by all means use an isolated environment. Wasn't
recommending against it, when needed.

Rather, that it is not a requirement for *every single script,* or even most.
As we've seen some folks have been scared into a heavy-handed solution.

> Perhaps it has to do with
> the decline of sys-admin skills over the years?
>
> Many millions of users of new Python users show up every year, using the
> language and its powerful ecosystem for data analytics and scientific computing,
> and they have no hope of having sys-admin skills.

"sys-admin" is a bit of an overstatement in my phrasing. The core is that you
need to understand how a PATH works and be able to run pip and cd into folders
and perhaps run rm/del, etc. Basic command-line skills?

In any case, if you can't do these, virtual envs are not a full solution either.

-Mike
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PWQDTGRQDQWPB6NBEVKDXVRECHQHYSSI/

[Wes Turner]

One aspect of conda packaging that's worth adopting is the 'test' block of the meta.yaml which contains commands necessary to *test* the built package.

You can add a pytest command to setup.py, but it's not as easy as:

https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml#L227-L291 :

```yaml

    test:

      downstreams:

        - cython

        - setuptools

      requires:

        - ripgrep

        - cmake

        - ninja

        - {{ compiler('c') }}

        # Tried to use enable_language(C) to avoid needing this. It does not work.

        - {{ compiler('cxx') }}

      files:

        - tests/distutils/*

        - tests/cmake/*

        - tests/cython/*

        - tests/prefix-replacement/*

      commands:

        - echo on  # [win]

        - set  # [win]

        - python -V

        - python3 -V            # [not win]

        - 2to3 -h

        - pydoc -h

        - python3-config --help  # [not win]

        - set "PIP_NO_BUILD_ISOLATION=False"  # [win]

        - set "PIP_NO_DEPENDENCIES=True"  # [win]

        - set "PIP_IGNORE_INSTALLED=True"  # [win]

        - set "PIP_NO_INDEX=True"  # [win]

        - set "PIP_CACHE_DIR=%CONDA_PREFIX%/pip_cache"  # [win]

        - set "TEMP=%CONDA_PREFIX%/tmp"  # [win]

        - mkdir "%TEMP%"  # [win]

        - python -Im ensurepip --upgrade --default-pip  # [win]

        # tzdata/zoneinfo test that will need the tzdata package to pass

        - python -c "from zoneinfo import ZoneInfo; from datetime import datetime; dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo('America/Los_Angeles')); print(dt.tzname())"

        - python -m venv test-venv

        - python -c "import sysconfig; print(sysconfig.get_config_var('CC'))"  # [not win]

        -  _CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu python -c "import sysconfig; print(sysconfig.get_config_var('CC'))"  # [linux64]

        # check for unreplaced @ symbols in sysconfig files

        - for f in ${CONDA_PREFIX}/lib/python*/_sysconfig*.py; do echo "Checking $f:"; if [[ `rg @ $f` ]]; then echo "FAILED ON $f"; cat $f; exit 1; fi; done  # [linux64 or osx]

        - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.a  # [unix]

        - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.nolto.a  # [unix]

        # https://github.com/conda-forge/python-feedstock/issues/384

        - if exist %PREFIX%\\Scripts\\pydoc exit 1  # [win]

        - if exist %PREFIX%\\Scripts\\idle exit 1  # [win]

        - if exist %PREFIX%\\Scripts\\2to3 exit 1  # [win]

        - if not exist %PREFIX%\\Scripts\\pydoc-script.py exit 1  # [win]

        - if not exist %PREFIX%\\Scripts\\idle-script.py exit 1  # [win]

        - if not exist %PREFIX%\\Scripts\\2to3-script.py exit 1  # [win]

        - if not exist %PREFIX%\\Scripts\\idle.exe exit 1  # [win]

        - if not exist %PREFIX%\\Scripts\\2to3.exe exit 1  # [win]

        - if not exist %PREFIX%\\Scripts\\pydoc.exe exit 1  # [win]

        - pushd tests

        -   pushd distutils

        -     python setup.py install -v -v

        -     python -c "import foobar"

        -   popd

        -   pushd prefix-replacement  # [unix]

        -     bash build-and-test.sh  # [unix]

        -   popd  # [unix]

        -   pushd cmake

        -     cmake -GNinja -DPY_VER={{ version }}

              # --trace --debug-output --debug-trycompile .

        -   popd

        - popd

        - test ! -f default.profraw   # [osx]

```

In which package you then store integration tests for your whole application and it's integrated dependencies is then up to you. Unfortunately, we often don't include tests/ in our packages, so it's literally not possible to run the tests in production even if you have pytest_requires installed.

Perhaps distros would do well to implement support for integration and per-package tests.

[Stephen J. Turnbull]

Mike Miller writes:

> "sys-admin" is a bit of an overstatement in my phrasing. The core
> is that you need to understand how a PATH works and be able to run
> pip and cd into folders and perhaps run rm/del, etc. Basic
> command-line skills?

That's what I would mean by basic sys-admin skills. And *surprise!*
my students don't have them, and don't need them ... until they start
using Python. They don't even understand installing packages in
conda.

If I had my druthers, we wouldn't have classes in Excel and
Powerpoint, and we wouldn't have classes in SPSS and SAS, and we
wouldn't even have classes in Python -- we'd have classes in envars,
registry, and shell scripts/bat files.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RJAZJLN6W76QONKD3K2UG6BPTOXPWE4D/

[Steven D'Aprano]

On Fri, Feb 26, 2021 at 11:41:56AM +0900, Stephen J. Turnbull wrote:
> Mike Miller writes:
>
> > "sys-admin" is a bit of an overstatement in my phrasing. The core
> > is that you need to understand how a PATH works and be able to run
> > pip and cd into folders and perhaps run rm/del, etc. Basic
> > command-line skills?
>
> That's what I would mean by basic sys-admin skills. And *surprise!*
> my students don't have them, and don't need them ... until they start
> using Python.

Is it *only* Python though? Wouldn't that be necessary if they were
learning Perl, Java, C, Ruby etc?



--
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OUU2BI6WYFFLJDAS3OCHJSFOATLNVADS/

[Oscar Benjamin]

On Fri, 26 Feb 2021 at 09:07, Steven D'Aprano <st...@pearwood.info> wrote:
>
> On Fri, Feb 26, 2021 at 11:41:56AM +0900, Stephen J. Turnbull wrote:
> > Mike Miller writes:
> >
> > > "sys-admin" is a bit of an overstatement in my phrasing. The core
> > > is that you need to understand how a PATH works and be able to run
> > > pip and cd into folders and perhaps run rm/del, etc. Basic
> > > command-line skills?
> >
> > That's what I would mean by basic sys-admin skills. And *surprise!*
> > my students don't have them, and don't need them ... until they start
> > using Python.
>
> Is it *only* Python though? Wouldn't that be necessary if they were
> learning Perl, Java, C, Ruby etc?

It is possible to teach students to do most things in Python without
using the command line. For example you can tell them all to install
anaconda and then use spyder as their editor. Or you can get them to
install and use vscode or pycharm or some other IDE with built in
virtualenv and git integration etc.

Where Python is awkward is at the point where you actually *want* to
teach students to use the command line. The problem is that Python is
not set up for command line use out of the box in a consistent way
across different platforms.

It would be nice to say: now open a terminal, type "python", and hit
the enter key and you'll see the Python prompt >>> but it's just not
that simple so the instructions end up being like:
"""
It might be possible to run python from the terminal by typing
"python". However that might not work or it might run the wrong
version/installation of Python. It might be that you haven't added
Python to your PATH environment variable or maybe you have depending
on which boxes you ticked when installing. Possibly you need to change
your PATH environment variable or possibly you should run Python as
"python3" or "py" instead but I have no way of saying without knowing
what OS you are on, where you installed Python from, what options you
selected in the installer, whether you have any other Python
installations etc. If you are on Windows and you get "permission
denied" then that's possibly because the MS-provided python.exe stub
is on your PATH so you first need to disable that in "app execution
aliases". Possibly you can't run Python from the normal terminal but
you can open a special terminal for Python that does have the PATH set
up such as the Anaconda Prompt.

Okay next lesson is how to use "pip" which is really easy: just type
"pip install X" but maybe you should actually type "pip3" or "python
-m pip" or "python3 -m pip" or "py -m pip" or actually you if you
installed Anaconda you might need to use "conda install X".
"""
When someone learns to use the command line for the first time it's
really quite helpful if you can tell them exactly what commands to
type. Knowing how to configure PATH and understanding what it does and
understanding about OS differences etc is potentially useful in the
long run but is a major distraction at the beginning.

There have been efforts to improve the situation but in some ways they
make it worse. For example the py launcher supposedly fixes some of
these problems on Windows but the fact that there is no analogue of it
on any non-Windows platform means that any instructions involving it
immediately need to split according to OS. Likewise the new python.exe
stub that ships with Windows is supposed to fix the Python PATH
problem but it actually *prevents* python from working in the terminal
if you have installed Python some other way (even if you have added it
to PATH):
https://stackoverflow.com/questions/56974927/permission-denied-trying-to-run-python-on-windows-10

The problems aren't only on Windows because you also have "python" vs
"python3" on OSX, Linux etc. On OSX you can also have conflicts
between the system Python and any user installed Python. On OSX some
installers set up the PATH for you by automagically editing
.bash_profile which can be helpful but can also be confusing. After
some time people don't actually know how to run the different versions
of Python that they installed through homebrew, from python.org or
from Anaconda etc and their .bash_profile is a mess. The question of
exactly what you should type to run python, pip, etc applies on Linux
although I find it less problematic just because Linux users typically
know how to use the terminal already.

For me as a user of Python I don't usually notice these problems
because I use some sort of virtual environment for everything so I can
always type "python" or "pip" and there are no conflicts between
different installations. When a student can't even get "python" to
work in the terminal though it's too much to ask them to start setting
up virtual environments.

--
Oscar

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/ZNMWRFR7PR7AISBUPBSE32WSNYZFUXCF/

[Paul Moore]

On Fri, 26 Feb 2021 at 13:31, Oscar Benjamin <oscar.j....@gmail.com> wrote:
> Where Python is awkward is at the point where you actually *want* to
> teach students to use the command line. The problem is that Python is
> not set up for command line use out of the box in a consistent way
> across different platforms.

My experience with Java is that it's at least as bad. (I'd say
"worse", but that might just be because I'm more familiar with Python
than with Java). For C, do I type cc, cl, gcc, clang, ...? Do I need
to start a "Visual Studio command prompt"? What if I prefer using
powershell? For Ruby and Perl, I have to install them, and assuming
that's easy, I've no idea what to do then.

I'm not sure what the point is here though. If people without command
line skills want to use the command line, they'll struggle? Well, yes,
that seems likely.

And I have no idea how this relates to virtual environments vs the
system Python.
Paul

So yes, Python's experience sucks, but so do most languages.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OL73XHKBP7Q3P2EZRHXMVXM4N6YAIDLE/

[Jim J. Jewett]

I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.

The problem is made worse because it often hits at the beginning instead of 7 lessons in, when they would have some reason to think it is worth struggling through and will eventually work.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/XWBEK2Q3DBY3M7HV7BN2GJPJCMPXJPQB/

[Oscar Benjamin]

On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett <jimjj...@gmail.com> wrote:
>
> I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.

Thanks Jim, that is the situation and yes they are not CS students.

The other point though is that it doesn't need to be like this. If the
issue was just installing Python and then setting up your PATH then
that's manageable. The problem is that even after doing those things
there isn't a "one way" to invoke Python from the command line. All
possible invocations (python, python3, py, ...) will fail for some
users. That's a problem for beginners but it's also a problem in any
situation where you want to write a command line that should run on
multiple platforms (e.g. in a Makefile or some other kind of script).

I see that the official Python tutorial now suggests typing
"python3.9" [1]. Is that what is recommended now? Obviously that would
fail for someone who had installed 3.8.

It would be great if Python could converge around a single way for
users to invoke the Python that they have installed and would want to
use. Python 2.x is beginning to fade away so python vs python3 makes
less sense now but maybe standardising on py for all platforms and
installs would be better (since py also has features for selecting
different versions and is not currently used for any "system python").

[1]: https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter


Oscar

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/HLMHW7U2UFVQ26NNUXDJGNII6EFDR7AX/

[Steven D'Aprano]

On Sat, Feb 27, 2021 at 01:04:08AM +0000, Oscar Benjamin wrote:
> On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett <jimjj...@gmail.com> wrote:
> >

> > I think his point is that most of his students (economics or
> > business, rather than comp sci) will never need to use Perl or C or
> > Java. Python is friendly enough to be useful, but this is still a
> > major pain point.
>
> Thanks Jim, that is the situation and yes they are not CS students.

Why is that relevant?

I'm not an automotive engineer, I haven't studied for an Automotive
Engineering Technology (AET) bachelor degree, but I still need to know
how to change a flat tire, fill the petrol tank with fuel, keep the
windscreen wiper water tank filled, put snow chains on if I live in a
region with snow, etc. Or pay somebody else to do it.

Python is not Scratch. It doesn't, and shouldn't, aim to be a
self-contained environment that isolates the user from their own
computer.

[Aside: the Scratch interpreter will actually refuse to run if it thinks
your code or documentation includes "bad words":

https://i.redd.it/nyqra8lmbei21.png

https://scratch.mit.edu/discuss/topic/32001/

https://scratch.mit.edu/discuss/topic/122410/

If this doesn't make you want to take off and nuke the entire planet
from orbit, then I don't know what's wrong with you. End of rant.]

It doesn't matter whether you are using Windows, Mac OS X or Linux,
whether your preferred UI is the command line or point-and-click, there
are some basic fundemental skills that every amateur or professional
programmer needs to know, such as:

- how to edit files and save them

- which file am I actually running?

- which interpreter am I actually running?

- how do I tell the computer to use a different interpreter?

These should be basic pre-requisites before learning to code, like being
able to read and write.


[...]

> I see that the official Python tutorial now suggests typing
> "python3.9" [1]. Is that what is recommended now? Obviously that would
> fail for someone who had installed 3.8.

It isn't elitist to expect that, at the barest minimum, any student of
programming should be able to adjust the command

python3.9 hello.py

to

python3.8 myscript.py

as required. If you can't debug

python3.9: command not found

or

can't open file '/home/steve/hello.py': [Errno 2] No such file or directory

then you're probably going to have a miserable time dealing with *hard*
bugs. Other more subtle errors may require a bit more knowledge, but
that's okay. We don't need to make programming a knowledge-free
activity.

> The other point though is that it doesn't need to be like this. If the
> issue was just installing Python and then setting up your PATH then
> that's manageable. The problem is that even after doing those things
> there isn't a "one way" to invoke Python from the command line. All
> possible invocations (python, python3, py, ...) will fail for some
> users. That's a problem for beginners but it's also a problem in any
> situation where you want to write a command line that should run on
> multiple platforms (e.g. in a Makefile or some other kind of script).

Writing code to run on any platform is a hard problem that requires
complex solutions. If you need to write a makefile that will run on a
million different flavours of Unix, including rare, old and weird ones,
then you have to expect that's a hard problem and you're probably going
to need to spend some effort getting it to work.

Not every problem is capable of being solved by a beginner with no
knowledge, and we don't have to bust our guts to make it an easy
problem. It's okay to delegate some problems to someone else, such as
make experts.

> It would be great if Python could converge around a single way for

> users to invoke the Python that they have installed ...

> py also has features for selecting different versions

I don't think that "py --someoption ..." is any better than the existing
solutions I already have for selecting different versions. Maybe on
Windows? I don't know how it works there.


--
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/PPVEFLOVZERTPRHBBAL4IYJHTCZJXOGN/

[Stephen J. Turnbull]

Steven D'Aprano writes:
> On Fri, Feb 26, 2021 at 11:41:56AM +0900, Stephen J. Turnbull wrote:

> > That's what I would mean by basic sys-admin skills. And *surprise!*
> > my students don't have them, and don't need them ... until they start
> > using Python.
>
> Is it *only* Python though? Wouldn't that be necessary if they were
> learning Perl, Java, C, Ruby etc?

I'm sure it is. But as I say *my* students are pretty naive, so they
only use languages supported by me (Python and R, but almost
exclusively Python) in Jupyter and installed via Conda. Not a lot of
folks around here can speak all of those, Japanese, and English, and
read minds in Chinese.[1] Otherwise they use site-licensed commercial
software (mostly SPSS, Mathematica, and office software).

Footnotes:
[1] This is the easiest part of all. It's not hard to guess what the
students are whispering in Chinese when you've just prohibited use of
Chinese in the seminar room. :-)

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/KHXH5D6LBSADHZ2NMIWWNUK2CE6IZ42Q/

[Stephen J. Turnbull]

Executive summary: This thread is about a "seven blind men and an
elephant" problem.

Jim J. Jewett writes:

> I think his point is that most of his students (economics or
> business, rather than comp sci) will never need to use Perl or C or
> Java. Python is friendly enough to be useful, but this is still a
> major pain point.

Not sure if "his" refers to me, but I don't know what they're going to
need to use. C/C++, maybe not, but there are a lot of Perl and
especially Java shops in Tokyo still. The largest group of my
students end up as what the Japanese call "SE" ("systems engineers"),
by which they mean a sales/requirements gathering/training/customer
relations role. Many of the rest end up in tech-adjacent roles. I
think all of them would benefit from a little command-line experience,
and a better understanding of how a group of programs cooperates to
become a system. As their teacher, I certainly would!

> The problem is made worse because it often hits at the beginning
> instead of 7 lessons in, when they would have some reason to think
> it is worth struggling through and will eventually work.

I think we're on the same page, but "7 lessons in" is still early in
my experience teaching Python. Languages where you have to explicitly
invoke a compiler and then run the executable, maybe you run into the
problem "at the beginning" (shades of Chapter 1 of _The C Programming
Language_!), but not with Python. Where my students run into problems
is when they *know* it was all worth it, because *now* they're ready
to use Python to do whatever it is they style "real work" -- and need
to install, maybe configure, and learn to use something that isn't an
included battery. Even in a well-curated framework like Anaconda,
"you can make a bottle adult-proof, but not child-proof".

So *my* point (I don't speak for anyone else) is pessimistic. I don't
think there's a one-size-fits-all solution to child-proofness. What
we need is children with better vocabularies.

From my point of view, what I want is to talk to students (and
colleagues!) about problems that I can often solve or install a
workaround in a few minutes. But when simple questions like "what
distribution did you install?" gets the answer "I just downloaded the
package. What's a distribution?", and the followup is, "well, where
did you get it?", "I clicked on the top link in Google", which leads
to "well, is it this one?" and "I can't tell...", it's very
frustrating for all concerned -- and we haven't even started on the
new package that isn't working.

When to them, it's pure fsckin' magic, and to me, it's muscle memory,
problem-solving across the gap is hard. For my purpose, cross-
platform distributions like Anaconda seem to be the best solution
available now and for the near future. But for many developers, they
have enough problems without dealing with the idea that some other
application has different version or configuration requirements for
some components.

I just don't see a world where any of the points of view are without
merit. People with more space than apps can put them *all* in venvs,
people with relatively simple "external" requirements can just use a
single installation, and people with the necessary skills can mostly
use the single installation and put a few "problem children" into
venvs of their own. Each of those groups will want Python itself to
take a different approach to the "problem".

So the question in the subject is valid and should be raised once in a
while, but the answer is always going to disappoint a majority. :-)

Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/COTOLXA4I76KMBPFQ2J5RQWNKKYCPRGE/

[Paul Moore]

On Sat, 27 Feb 2021 at 01:08, Oscar Benjamin <oscar.j....@gmail.com> wrote:
>
> On Fri, 26 Feb 2021 at 23:06, Jim J. Jewett <jimjj...@gmail.com> wrote:
> >
> > I think his point is that most of his students (economics or business, rather than comp sci) will never need to use Perl or C or Java. Python is friendly enough to be useful, but this is still a major pain point.
>
> Thanks Jim, that is the situation and yes they are not CS students.

I've been in this situation with people who are professional
developers (but in environments where "development" is done in a
self-contained application IDE). I completely understand and
sympathise with the idea that suddenly having to learn command line
skills when all you'd signed up for was to learn Python is both
daunting and off-putting.

> The other point though is that it doesn't need to be like this. If the
> issue was just installing Python and then setting up your PATH then
> that's manageable. The problem is that even after doing those things
> there isn't a "one way" to invoke Python from the command line. All
> possible invocations (python, python3, py, ...) will fail for some
> users. That's a problem for beginners but it's also a problem in any
> situation where you want to write a command line that should run on
> multiple platforms (e.g. in a Makefile or some other kind of script).

This is 100% true. (At least the part where you say that "it's a
problem". I'm not sure if "it doesn't need to be like this" is true -
can you point to a language where the command line "just works" the
way you suggest?)

> I see that the official Python tutorial now suggests typing
> "python3.9" [1]. Is that what is recommended now? Obviously that would
> fail for someone who had installed 3.8.
>
> It would be great if Python could converge around a single way for
> users to invoke the Python that they have installed and would want to
> use. Python 2.x is beginning to fade away so python vs python3 makes
> less sense now but maybe standardising on py for all platforms and
> installs would be better (since py also has features for selecting
> different versions and is not currently used for any "system python").

It *would* be great. I don't think there's anyone who disagrees with
this point. But just thinking "it would be great" isn't of much
practical use. People have been trying to fix this issue for many
years (quite apart from all of the core Python debates, it's an
endless, frustrating topic in the packaging community - I can't tell
you how many times we've tried to come up with a good way to tell
people how to invoke pip).

In my view, for people learning Python in a context where command line
skills are *not* the norm, it would be ideal to focus on one (or more)
"integrated environment", in much the same way as no-one learns
command-line R, they all use R Studio. Unfortunately, the Python
community isn't good at getting behind a single solution like this, so
we have IDLE, Jupyter, Spyder, and probably a host of others.

The command is *hard*. People used to it tend to forget that. I know
people who are good developers (in an IDE) who struggle with how to
specify options to commands, and how to quote arguments. (Not
surprising, as both of those depend on what OS and what shell you're
using, as well as being entirely learned information - there's no
"intuitive" reason why options start with a dash, or you quote with
double quotes not single quotes in Windows). I don't think we should
be pointing beginners at the command line *at all*. But as a
community, we don't really have any better option. (Individual
trainers or organisations can make their own choices, but that's not
easy in the absence of a community consensus).

Paul

PS Having a standard "integrated environment" that people learn in has
its own problems, of course, as we risk training a new generation of
Python users who don't know how to use their Python code outside of
that IDE, and who therefore can't work with git or automated tests,
etc, etc... I'm looking at Java and the "if my IDE doesn't do it, I'm
lost" mindset here. I don't know if R has fallen into a similar trap
with R Studio.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/3FEHMUITOB4CW3TFQKFBAS2AEEXJEUXE/

[Oscar Benjamin]

On Sat, 27 Feb 2021 at 11:04, Paul Moore <p.f....@gmail.com> wrote:
>
> On Sat, 27 Feb 2021 at 01:08, Oscar Benjamin <oscar.j....@gmail.com> wrote:
> >
> > The other point though is that it doesn't need to be like this. If the
> > issue was just installing Python and then setting up your PATH then
> > that's manageable. The problem is that even after doing those things
> > there isn't a "one way" to invoke Python from the command line. All
> > possible invocations (python, python3, py, ...) will fail for some
> > users. That's a problem for beginners but it's also a problem in any
> > situation where you want to write a command line that should run on
> > multiple platforms (e.g. in a Makefile or some other kind of script).
>
> This is 100% true. (At least the part where you say that "it's a
> problem". I'm not sure if "it doesn't need to be like this" is true -
> can you point to a language where the command line "just works" the
> way you suggest?)

Python from 15 years ago :)

Maybe I'm misremembering but when I first started with Python there
didn't seem to be any ambiguity about what to type in the terminal.
The first split as I remember it was on Linux when python3 came along
but python kept referring to 2.x on (most) Linux. At the same time
python3 was not added to Windows and instead python became 3.x. Then
there was the py launcher for Windows. Perhaps this problem always
existed on OSX although I didn't use it at all at the time.

More recently I've been exploring Julia. You could compare these pages:

https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter
https://docs.julialang.org/en/v1/manual/getting-started/

When installing julia you select the installer for your OS and there
are some instructions for setting up PATH (although I didn't need to
do that manually on OSX). After setup the instruction is that you type
"julia" in the terminal and then Ctrl-D to exit. There are no caveats
or mentions of different operating systems or words like "if" and
"usually". The setup process is platform specific but then the usage
instructions are not.

--
Oscar

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/BTDEIUAVZN2GWRVQY34FJHRF6YOWKZKT/

[Steven D'Aprano]

On Sat, Feb 27, 2021 at 12:56:17PM +0000, Oscar Benjamin wrote:
> On Sat, 27 Feb 2021 at 11:04, Paul Moore <p.f....@gmail.com> wrote:

> > This is 100% true. (At least the part where you say that "it's a
> > problem". I'm not sure if "it doesn't need to be like this" is true -
> > can you point to a language where the command line "just works" the
> > way you suggest?)
>
> Python from 15 years ago :)

Fifteen years ago, if I remember correctly, I had Python 1.5, 2.4 and
2.5 installed. (I may have some of the versions mixed up.)

When I typed `python` at the command line, the OS read the faint
electrical currents at my finger tips through the keyboard, extrapolated
the state of my brain, predicted the interpreter I wanted, and
unfailingly guessed the right one.

Nah just kidding. I'm pretty sure it ran 2.4, always.

> When installing julia you select the installer for your OS and there
> are some instructions for setting up PATH (although I didn't need to
> do that manually on OSX). After setup the instruction is that you type
> "julia" in the terminal and then Ctrl-D to exit. There are no caveats
> or mentions of different operating systems or words like "if" and
> "usually". The setup process is platform specific but then the usage
> instructions are not.

So what happens when I have two different Julia versions
installed?


--
Steve

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/T3HHANHVUZOM67TA72R6LZFTMOZHJ2DM/

[Paul Moore]

On Sat, 27 Feb 2021 at 13:03, Oscar Benjamin <oscar.j....@gmail.com> wrote:
> More recently I've been exploring Julia. You could compare these pages:
>
> https://docs.python.org/3/tutorial/interpreter.html#invoking-the-interpreter
> https://docs.julialang.org/en/v1/manual/getting-started/
>
> When installing julia you select the installer for your OS and there
> are some instructions for setting up PATH (although I didn't need to
> do that manually on OSX). After setup the instruction is that you type
> "julia" in the terminal and then Ctrl-D to exit. There are no caveats
> or mentions of different operating systems or words like "if" and
> "usually". The setup process is platform specific but then the usage
> instructions are not.

Sorry, I couldn't resist replying to this.

I went to the Julia page. That linked me to the download page. Which
offered me multiple versions. Which should I take? Stable, LTS or
"upcoming"? As a beginner without the experience I have hitting stuff
like this, how would I know? Let's assume "stable". OK, download and
run an installer is easy enough for me on Windows, but let's say I'm
doing this on Linux, I get the choice of "Generic Linux on x86" or
"Generic Linux on ARM". Huh? I'm on Ubuntu, not "Generic", so I'm
confused. Then I have "64-bit (GPG)" or "64-bit (musl) (GPG)" Or AArch
if I chose "ARM". How the heck would I know. Any of those get me a
.tar.gz file, which I don't know how to use...

Going to the Linux help, I get told to use wget, not the file I just
downloaded. And then I unpack it, and now I'm into setting PATH to
/tmp/julia-something/bin (because I did this in /tmp, as I have no
idea where I want to put this long term).

OK, my linux thought experiment isn't going well. Let's go back to
Windows, where I was lucky enough to pick "installer" rather than
"portable", so I avoided a world of pain about where to put the
package, just like I hit on Linux. The installer puts Julia in my
personal apps directory, just like Python does. I'm going to do
something "advanced" here ;-) and deselect "desktop shortcut" because
I hate clutter on my desktop. OK, it's complete, let's select "run
julia". That got me a prompt. Cool. Start menu item does the same.
Also cool.

BUT... The python installer on windows is *exactly the same*. Installs
to user's area, start menu entry that runs the REPL. So that's no
worse.

Suppose I go to a command prompt on my own and try to run Julia. Open
a terminal, "julia". No good, I need to set PATH. Hmm, wasn't there
something in the help on the Julia website about that? Off we go to
check. Weren't we here before when we were using Python? And the
instructions on the Julia website are about as complex as I'd expect,
and much the same as for Python. So still no worse for Python. But the
Python installer has "add Python to PATH" that does this stuff for you
(not selected by default, because of reasons that you may or may not
agree with, but at least it's *there*). And there's the py launcher,
but let's ignore that, as it works round some complexity at the cost
of some different complexity, so it's a distraction at this stage.

Honestly, I think Julia is overall slightly *worse* than Python (at
least on Windows). But it's almost identical in practice.

The one thing that *is* substantially worse for Python, is the
circumlocutions needed in the documentation to say how to run Python.
But that's 100% down to us not being willing to say "just type the
command python". And the reason for *that* is mostly historical,
related to the Python 3 transition and what happened on Linux over the
python/python2/python3 commands, and to a lesser extent the
introduction of the launcher on Windows ("lesser", because using the
launcher on the command line wasn't recommended until some time after
its introduction, at which point the python2/python3 split was
established)...

Paul

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/7AVAVWMGNRDLAEQHDLWWTPHMLJG4V7RD/

[Jim J. Jewett]

[They are not CS students]

> Why is that relevant?

Because for many users, python is NOT a programming language; it is an application like any other. It happens to be very powerful and flexible, but the point isn't to program; it is to produce better reports.

If the hurdle to get started (either initially, or after a while away) is too high, then it is not a useful tool. It is OK to have all sorts of power-user knobs, but the default instructions should just work ... they shouldn't have to say "ensure you have proper settings for these 6 things you don't care about or understand", because the default install should just choose reasonable default settings and take care of them.

> there are some basic fundemental skills that every amateur or
> professional programmer needs to know, such as:

My point is that plenty of python users have no intention of being a programmer.

> how to edit files and save them

(fair ... but plenty of programs don't even *let* you do this with an external editor; there is no reason the _default_ should force people to pick and configure an external editor. IDLE tends to do OK here.)

> which file am I actually running?

> which interpreter am I actually running?

> how do I tell the computer to use a different interpreter?

If you need to care about any of these, then the environment is fighting you -- and the application probably stinks. Programmers have to deal with it because of bootstrapping, but there is no reason that we should assume all python users need that flexibility or want that responsibility.

> Writing code to run on any platform is a hard problem that
> requires complex solutions.

Thus the preference for solving it once in the library/official documentation/reference installer. Experts can build on it or opt out, but non-programmers shouldn't have to worry about cross-platform issues just to use python.

-jJ

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/FPPE35KUME5JDDRSZ44RXRDN366EJNAC/

[Nick Coghlan]

On Wed, 24 Feb 2021 at 10:49, Random832 <rand...@fastmail.com> wrote:
>
> I was reading a discussion thread <https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e> about various issues with the Debian packaged version of Python, and the following statement stood out for me as shocking:
>
> Christian Heimes wrote:
> > Core dev and PyPA has spent a lot of effort in promoting venv because we don't want users to break their operating system with sudo pip install.
>
> I don't think sudo pip install should break the operating system. And I think if it does, that problem should be solved rather than merely advising users against using it. And why is it, anyway, that distributions whose package managers can't coexist with pip-installed packages don't ever seem to get the same amount of flak for "damaging python's brand" as Debian is getting from some of the people in the discussion thread? Why is it that this community is resigned to recommending a workaround when distributions decide the site-packages directory belongs to their package manager rather than pip, instead of bringing the same amount of fiery condemnation of that practice as we apparently have for *checks notes* splitting parts of the stdlib into optional packages? Why demand that pip be present if we're not going to demand that it works properly?

The reason venv is promoted as heavily as it is is because it's the
only advice that can be given that is consistently correct regardless
of the operating system the user is running locally, whereas safely
using a system-wide Python installation varies a lot depending on
whether you're on Windows, Mac OS X, or Linux (let alone some other
platform outside the big 3 desktop clients).

conda is also popular for the same reason: while the instructions for
installing conda in the first place are OS-dependent, once it is up
and running you can use consistent platform independent conda commands
rather than having to caveat all your documentation with
platform-specific instructions.

Apple moved all of their dynamic language interpreter implementations
to inaccessible-by-default locations so Mac OS X users would stop
using them to run their own code.

Alongside that, we *have* worked with the Linux distro vendors to help
make "sudo pip install" safe (e.g [1]), but that only helps if a user
is running a new enough version of a distro that has participated in
that work.

However, while the option of running "platform native" environments
will never go away, and work will continue to make it less error
prone, the level of knowledge of your specific OS's idiosyncrasies
that it requires is almost certainly going to remain too high for it
to ever again become the default recommendation that it used to be.

Cheers,
Nick.

[1] https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe (Note:
this change mitigated some aspects of the problem in a way similar to
what Debian does, but still doesn't solve it completely, as custom
Python builds may still make arbitrary changes)

P.S. "But what about user site-packages?" you ask. Until relatively
recently, Debian didn't put the user's local bin directory on the
system path by default, so commands provided by user level package
installs didn't work without the user adjusting their PATH. The
CPython Windows installer also doesn't adjust PATH by default (for
good reasons). And unlike a venv, "python -m" doesn't let you ensure
that the code executed is the version installed in user site-packages
- it could be coming from a directory earlier in sys.path.

--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/RDLEH6DUF57UB6U4HNL2QRVAJY4KDSSJ/

[Stephen J. Turnbull]

Jim J. Jewett writes:

> > which file am I actually running?
> > which interpreter am I actually running?
> > how do I tell the computer to use a different interpreter?
>
> If you need to care about any of these, then the environment is
> fighting you -- and the application probably stinks. Programmers
> have to deal with it because of bootstrapping, but there is no
> reason that we should assume all python users need that flexibility
> or want that responsibility.

Be careful! I agree about need for flexibility and desire for
responsibility, but in fact these problems present themselves
frequently when helping people with their work. "Which file am I
actually [edit]ing?" is one that comes up in just about every meeting:
documents are undated (or dated with the deadline date), people are
looking at different versions, and not infrequently the presenter
themself has a Documents/Downloads confusion and gets it wrong. Then
"where is the one I want to be editing?" requires an oracle. :-)
"Which program am I running?" doesn't come up that often, but it does
come up when helping people with "doing something", especially trying
to find menu items across major versions. Interestingly enough, "how
do I tell the computer to use a different interpreter?" doesn't come
up for me outside of Python (and even there rarely) -- everybody who
consults me knows how to use "Open With".

So I agree with the basic principle that programs should Just Work by
default, but it's not always possible. Eg, there's good reason why
Downloads and Documents both exist. So it helps to have users know
something about implementations and sysadminning stuff, especially
when remote troubleshooting.

The question for Python is given the history and the plethora of ways
to invoke Python and find packages and programs, can we do better with
a minimum of backward incompatibility? Humans are generally better at
learning this stuff than bags of bits on spinning platters are!

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/OAYWDLL26YTSIFTKGL6LWP7TU4PXUH6K/

[Stephen J. Turnbull]

Paul Moore writes:

> The one thing that *is* substantially worse for Python, is the
> circumlocutions needed in the documentation to say how to run Python.
> But that's 100% down to us not being willing to say "just type the
> command python". And the reason for *that* is mostly historical,
> related to the Python 3 transition

I don't think that's entirely true, and the roots are much earlier
than Python 3. We had the same problem with Linux distributions
(especially Red Hat) and version 1 vs. version 2. This was crucial
for those of us handling multibyte languages: getting Unicode Inside,
even in the non-str form of Python 2, really changed things. But
Python 2.0 was a .0 release. For a lot of developers dealing with
unibyte languages, 1.5.2 was the workhorse until about Python 2.2.

> and what happened on Linux over the python/python2/python3
> commands, and to a lesser extent the introduction of the launcher
> on Windows

I don't know about Windows. It might have been possible to arrange
that just "python" was the right way to do things, and have "-2" and
"-3" flags for specifying compatibility. But it just wasn't possible
on *nix systems because /usr/bin/python was typically used for system
software. They shouldn't have done that (they should have kept in it
in an OS-specific place and used full shebangs), but that's not how it
went ....

I think those are important things to keep in mind when comparing with
a very young language like Julia (Julia didn't make *any* backward
compatibility promises until 1.0 in August 2018). A lot of Python's
problems occur *because* of its multi-use potential and *because* of
its wide adoption in practice.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/S7H6W75N62YGW5L2WYQH53BTFDJJWRZK/

[Jim J. Jewett]

> And unlike a venv, "python -m" doesn't let you ensure
> that the code executed is the version installed in user site-packages

I have had enough problems with this that when I do modify/replace something, I put in a marker that I can check for explicitly. Expecting this marker to run automatically at import (and the results not to be swallowed) turned out to be insufficient.

> it could be coming from a directory earlier in sys.path.

The obvious solution is to put the current directory in front of sys.path.
Alas, security folks didn't like that idea.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/M3U72J6OR53LF5REA7LOZBIQUUXXHV65/

[Oscar Benjamin]

On Sun, 28 Feb 2021 at 07:04, Stephen J. Turnbull
<turnbull....@u.tsukuba.ac.jp> wrote:
>
> Jim J. Jewett writes:
>
> > > which file am I actually running?
> > > which interpreter am I actually running?
> > > how do I tell the computer to use a different interpreter?
> >
> > If you need to care about any of these, then the environment is
> > fighting you -- and the application probably stinks. Programmers
> > have to deal with it because of bootstrapping, but there is no
> > reason that we should assume all python users need that flexibility
> > or want that responsibility.
>

<snip>

>
> So I agree with the basic principle that programs should Just Work by
> default, but it's not always possible. Eg, there's good reason why
> Downloads and Documents both exist. So it helps to have users know
> something about implementations and sysadminning stuff, especially
> when remote troubleshooting.
>
> The question for Python is given the history and the plethora of ways
> to invoke Python and find packages and programs, can we do better with
> a minimum of backward incompatibility? Humans are generally better at
> learning this stuff than bags of bits on spinning platters are!

I think that the py launcher is actually a good solution to many of
these problems. It can also be improved but the main reason it doesn't
help right now is just the fact that it's only for Windows and also
isn't always installed on Windows. When the launcher is installed
though then:

- It is on PATH
- It can list the Python installations with "py -0p"
- It can be used to select a particular Python version/installation to
use from the command line e.g. "py -3.8 myscript.py"
- It is possible to configure a default version (although I think you
have to do it with an environment variable)

There are some ways that the launcher situation could be improved though:

- I think that the launcher is only installed in an all users install.
There would need to be a way to do it in user-space as well (even if
it means the user has to configure PATH).
- Listing installations with "py -0p" is somewhat cryptic
- It would be better if you could use the launcher itself to set the
default Python e.g. "py --set-default-python=3.8"
- The launcher isn't installed by Anaconda

On the last point I think that although Anaconda doesn't install the
launcher you can use the launcher to run the python from the Anaconda
installation. Ideally Anaconda would install the launcher but it could
also be possible to install it separately.

If py was provided on OSX, Linux etc along with the very clear
guidance that it is supposed to be for "users" rather than the
"system" then maybe that could be a better situation. It would be
backwards compatible since the launcher already exists on Windows and
"py" hasn't been used for anything on non-Windows platforms (AFAIK).


--
Oscar

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/LZGV4DDFZKA4NYQVR6JVNG4HVBXDRZFZ/

[Wes Turner]

Is there a tool that (1) detects import name collisions; and (2) attempts to read package metadata and package file checksums (maybe from the ZIP 'manifest')?

In order to:

- troubleshoot module shadowing issues

  - $PATH

  - sys.path

    - `python -m site`

  - incomplete and overlapping uninstallations:

pip install a

pip install a_modified # pip uninstall a?

pip install pdbpp

pip uninstall a_modified

ls -altr "${site-packages[*]}" 

strace -e trace=file python -c 'import pdb'

****

When shouldn't site customizations be added to the site module?

https://docs.python.org/3/library/site.html

When should customizations be built into the build instead of a runtime conditional?

[Eryk Sun]

On 2/28/21, Oscar Benjamin <oscar.j....@gmail.com> wrote:
>
> - It is possible to configure a default version (although I think you
> have to do it with an environment variable)

The py launcher in Windows supports a "py.ini" file beside the
executable and in %LocalAppData%. The equivalent of the PY_PYTHON,
PY_PYTHON2, and PY_PYTHON3 environment variables can be set in the
"[defaults]" section as "python", "python2", and "python3" settings.

The ini file also supports a "[commands]" section to define additional
virtual commands for shebangs. Regular filepaths are also supported in
shebangs -- e.g. #!"path\to\any\program.exe".

> - I think that the launcher is only installed in an all users install.

It defaults to an all-users install, but it can also be installed for
just the current user in "%LocalAppData%\Programs\Python\Launcher". In
this case, the installation directory always gets added to PATH.

> - Listing installations with "py -0p" is somewhat cryptic

There's also the long-form options `--list` and `--list-paths`.

> - It would be better if you could use the launcher itself to set the
> default Python e.g. "py --set-default-python=3.8"

It's pretty simply to run `set PY_PYTHON=3.8`, and persist the value
in the registry with `setx.exe PY_PYTHON 3.8`. (But don't use setx.exe
naively to set PATH.)

> On the last point I think that although Anaconda doesn't install the
> launcher you can use the launcher to run the python from the Anaconda
> installation.

I don't use Anaconda, but I don't think that's supposed to be the case
according to PEP 514. The launcher only looks for PSF development
distributions in the "PythonCore" registry key.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/JPQLBEYVWV2XGXO44JJ4YPN3KFJJYN2Q/

[Oscar Benjamin]

On Mon, 1 Mar 2021 at 00:13, Eryk Sun <ery...@gmail.com> wrote:
>
> On 2/28/21, Oscar Benjamin <oscar.j....@gmail.com> wrote:
> >
> > - It is possible to configure a default version (although I think you
> > have to do it with an environment variable)
>
> The py launcher in Windows supports a "py.ini" file beside the
> executable and in %LocalAppData%. The equivalent of the PY_PYTHON,
> PY_PYTHON2, and PY_PYTHON3 environment variables can be set in the
> "[defaults]" section as "python", "python2", and "python3" settings.
>
> The ini file also supports a "[commands]" section to define additional
> virtual commands for shebangs. Regular filepaths are also supported in
> shebangs -- e.g. #!"path\to\any\program.exe".

Yes, I forgot about that. I still think it would be better if you
could do it with the py command...

> > - I think that the launcher is only installed in an all users install.
>
> It defaults to an all-users install, but it can also be installed for
> just the current user in "%LocalAppData%\Programs\Python\Launcher". In
> this case, the installation directory always gets added to PATH.

Oh, okay. So does that mean that it's always on PATH unless the user
*explicitly unticks* the "install the launcher" box for both single
user and all user installs?

> > - It would be better if you could use the launcher itself to set the
> > default Python e.g. "py --set-default-python=3.8"
>
> It's pretty simply to run `set PY_PYTHON=3.8`, and persist the value
> in the registry with `setx.exe PY_PYTHON 3.8`. (But don't use setx.exe
> naively to set PATH.)

Those commands aren't portable to any other OS though. The advantage
of having it done through py is that it can (hopefully) be made
consistent across platforms.

> > On the last point I think that although Anaconda doesn't install the
> > launcher you can use the launcher to run the python from the Anaconda
> > installation.
>
> I don't use Anaconda, but I don't think that's supposed to be the case
> according to PEP 514. The launcher only looks for PSF development
> distributions in the "PythonCore" registry key.

Maybe I misunderstood the situation. If py can only be used for PSF
binaries then that wouldn't work. I guess that can be changed
though...


--
Oscar

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/VDB4RXS3RUO6ALU4S22N6PKWTHIHWI2K/

[Eryk Sun]

On 2/28/21, Oscar Benjamin <oscar.j....@gmail.com> wrote:
>
> Oh, okay. So does that mean that it's always on PATH unless the user
> *explicitly unticks* the "install the launcher" box for both single
> user and all user installs?

If the launcher gets installed, it will be available in PATH. IIRC,
the installer only allows installing the launcher for the current user
if it is not installed already for all users. Thus only one version
should ever exist in PATH, which is set in either the user or system
"PATH" value in the registry, but not both.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/3QWZRQSI5BTM3GQRWNH2UQN4L7RONAPJ/