[Python-ideas] Use 'bin' in virtual environments on Windows

[Svein Seldal]

Using py in the three OS-es (unix-like, Mac and Win) has become very
similar in the last years, which is great. Making portable py code has
become very easy. However, there is one peculiarity in Win which I find
annoying: virtual environments insists on placing executables in the
`Scripts` directory, while the other platforms use `bin`. This forces
portable scripts and programs on the outside of py to be OS-aware in
order to interact with a venv on a Win-system vs other systems.

I would like to proposing the ability to configure virtual environments
to use `bin` directory for executable on Windows. Personally I think it
would be smart move if it were changed to `bin` as default, making all
py platform act consistently. However, I do understand that there might
be existing code out there that depend on using `Scripts`.

Best regards,
Svein
_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python...@python.org/message/5OFYP734PMXBVIXEW444IU4CPUI7KTIB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Paul Moore]

There is a *lot* of code that depends on the Windows directory being "Scripts". There have been proposals in the past to change it (sorry, no links but it has come up a few times on the various packaging forums), but it simply breaks too much for too little gain.

If you're serious about wanting this, and the gain to you is sufficient to justify the work, I'd suggest reviewing as many packaging tools as you can find (virtualenv, venv, pip, setuptools, pipenv, pipx, poetry, hatch, pew, as well as core Python in the sysconfig module and distutils would be a good starting point). Work out what changes would be needed to all of these to support using "bin" on Windows, being sure to work out how to handle backward compatibility - people will still have environments that use "Scripts" for many years to come, even if we change everything, and most of the tools I mentioned need to support Python back to at least version 3.7, if not earlier. Even after doing all of that, you'd likely *still* have a huge issue to address, which is all of the various personal and in-house utilities that never get published anywhere, but which either hard code the "if Windows, then Scripts else bin" test, or simply use whichever value is appropriate for the platform they run on (there may be no need for portability anyway).

A practical approach may be to develop some form of library that "hides" the difference behind some form of API for finding the correct value, get that added to the stdlib and wait a few years until it's adopted everywhere (because it's so well-designed and convenient ;-)) Then, you can change the location. But at that point no-one will care, because they don't ever reference the actual location, they just use the API anyway :-)

Paul

[Simão Afonso]

On 2022-07-21 09:55:21, Paul Moore wrote:
> A practical approach may be to develop some form of library that "hides"
> the difference behind some form of API for finding the correct value, get
> that added to the stdlib and wait a few years until it's adopted everywhere
> (because it's so well-designed and convenient ;-)) Then, you can change the
> location. But at that point no-one will care, because they don't ever
> reference the actual location, they just use the API anyway :-)

I think an option when creating the "venv" (something like "--bin-name
bin", or "--use-bin") is enough. Whatever creates the environment also
handles the fact that "bin" is on the "right" location now.
If you don't control the venv creation, you just use the default,
or accept a similar option.

I would vote against having a environment variable or some kind of
global setting, that will break backwards compatibility.

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/ZCNTOSI4IHEX52I5TVB6QOF3VI4OX7EB/

[Thomas Grainger]

> A practical approach may be to develop some form of library that "hides"
the difference behind some form of API for finding the correct value, get
that added to the stdlib and wait a few years until it's adopted everywhere
(because it's so well-designed and convenient ;-)) Then, you can change the
location.

Is this the canonical location of this information?

https://github.com/python/cpython/blob/3.10/Lib/sysconfig.py#L56

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/FCB675YE3TJEXUU65ZVV5QSJHFDQDJFY/

[Paul Moore]

On Thu, 21 Jul 2022 at 11:07, Simão Afonso <simao....@powertools-tech.com> wrote:

On 2022-07-21 09:55:21, Paul Moore wrote:
> A practical approach may be to develop some form of library that "hides"
> the difference behind some form of API for finding the correct value, get
> that added to the stdlib and wait a few years until it's adopted everywhere
> (because it's so well-designed and convenient ;-)) Then, you can change the
> location. But at that point no-one will care, because they don't ever
> reference the actual location, they just use the API anyway :-)

I think an option when creating the "venv" (something like "--bin-name
bin", or "--use-bin") is enough. Whatever creates the environment also
handles the fact that "bin" is on the "right" location now.
If you don't control the venv creation, you just use the default,
or accept a similar option.

How would that work? Would the value of bin-name be stored somewhere and then all tools would need to refer to that rather than just selecting based on platform like now? You'd still need to change all the tools, or your choice of directory simply wouldn't make any difference...

Paul

[Paul Moore]

On Thu, 21 Jul 2022 at 11:33, Thomas Grainger <tag...@gmail.com> wrote:

> A practical approach may be to develop some form of library that "hides"
the difference behind some form of API for finding the correct value, get
that added to the stdlib and wait a few years until it's adopted everywhere
(because it's so well-designed and convenient ;-)) Then, you can change the
location.

Is this the canonical location of this information?

https://github.com/python/cpython/blob/3.10/Lib/sysconfig.py#L56

In theory, yes. In practice, if that worked, we wouldn't get people asking about changing this in the first place... Non-Python programs can get the script location from sysconfig using

    py -c "import sysconfig; print(sysconfig.get_path('scripts'))"

But yes, it's likely that sysconfig *is* that API - it's just that for whatever reason, people haven't adopted it well enough that we can afford to change the location without breaking things. Which makes this more of a social problem than a technical one (get people to use sysconfig and the problem goes away).

Paul

[Simão Afonso]

On 2022-07-21 12:00:18, Paul Moore wrote:
> How would that work? Would the value of bin-name be stored somewhere and
> then all tools would need to refer to that rather than just selecting based
> on platform like now? You'd still need to change all the tools, or your
> choice of directory simply wouldn't make any difference...

Right, I wasn't thinking about all the other tools like poetry and the
like.

I though just changing "venv" and "pip" was enough.

Maybe an option in "pyproject.toml", which all tools would consume?

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/RR7TEJSGOQOYLGAQ2PZRYGO5KFQE7ZED/

[Thomas Grainger]

I'd recommend writing a virtualenv plugin that configures https://virtualenv.pypa.io/en/latest/extend.html#virtualenv.discovery.discover.Discover

Overriding the interpreter install path here:

https://github.com/pypa/virtualenv/blob/6f8fb11aed7c05a535c30040ce314ae5eec1dcc7/src/virtualenv/create/describe.py#L27

And see what you can make work and what doesn't work

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/G4KCMMEMIFHGBWDV7RXDQWHST4ZXSSMC/

[Svein Seldal]

On 2022-07-21 13:05, Paul Moore wrote:
> On Thu, 21 Jul 2022 at 11:33, Thomas Grainger <tag...@gmail.com>
>>

>> Is this the canonical location of this information?
>>
>> https://github.com/python/cpython/blob/3.10/Lib/sysconfig.py#L56
>
> In theory, yes. In practice, if that worked, we wouldn't get people
> asking about changing this in the first place... Non-Python programs
> can get the script location from sysconfig using
>
> py -c "import sysconfig; print(sysconfig.get_path('scripts'))"
>
> But yes, it's likely that sysconfig *is* that API - it's just that for
> whatever reason, people haven't adopted it well enough that we can
> afford to change the location without breaking things. Which makes
> this more of a social problem than a technical one (get people to use
> sysconfig and the problem goes away).

I ran a grep Scripts in a freshly created venv and I do get quite many
hits. Even pip is hard coded:

https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/_distutils.py#L145-L148
https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/entrypoints.py#L48

It also creates files which seems to come from Py 3 standard lib.

venv/Lib/site-packages/pip/_vendor/distlib/_backport/sysconfig.cfg

And this is just considering pip and setuptools...

I see now that this proposal is futile, and I'm backing down. I'd wish
it thou, but we're too committed to the existing design choices and that
ship has sailed.


Best regards,
Svein

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/MBXSULEYVPWUCJ7EW5NQBWJQLJLXJCPZ/

[Simão Afonso]

On 2022-07-21 17:04:16, Svein Seldal wrote:
> https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/_distutils.py#L145-L148

This actually works great! If the "Scripts" folder does not exist, it
uses "bin".

> https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/entrypoints.py#L48

This is a just a small patch anyway.

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/JZZ6QJVEJBC4TZF6SUBWD3TUK2RHIRTW/

[Christopher Barker]

I still am dumbfounded that this wasn’t platform I dependent in the first place, but you know what essay about hindsight.

However, I’m no Windows expert, but I *think* the modern Windows file system(s?) support something like symlinks. It’s an under-the-hood feature, but maybe it’s possible to add a symlink for bin. 

Maybe I’m wrong, and/or it’s not possible on all file systems Python needs to support, in which case *nix systems do support linking, so we could support “Scripts” on all systems.

Just a thought. 

-CHB

On Thu, Jul 21, 2022 at 8:26 AM Simão Afonso <simao....@powertools-tech.com> wrote:

On 2022-07-21 17:04:16, Svein Seldal wrote:

> https://github.com/pypa/pip/bob/main/src/pip/_internal/locations/_distutils.py#L145-L148

This actually works great! If the "Scripts" folder does not exist, it
uses "bin".

> https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/entrypoints.py#L48

This is a just a small patch anyway.
_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python...@python.org/message/JZZ6QJVEJBC4TZF6SUBWD3TUK2RHIRTW/
Code of Conduct: http://python.org/psf/codeofconduct/

--

Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython

[Paul Moore]

On Thu, 21 Jul 2022 at 16:45, Christopher Barker <pyth...@gmail.com> wrote:

I still am dumbfounded that this wasn’t platform I dependent in the first place, but you know what essay about hindsight.

Indeed. I've no idea why we had the difference in the first place, but that's water under the bridge at this point.

 

However, I’m no Windows expert, but I *think* the modern Windows file system(s?) support something like symlinks. It’s an under-the-hood feature, but maybe it’s possible to add a symlink for bin.

Maybe. But symlinks are not available on all Windows systems (they need certain settings enabled) so we need to be able to work without them (the same is true on Unix, of course, as symlink support requires the filesystem to support it, and not all filesystems do, - but that's not relevant here).

The long and short of it is that for a significant amount of time, the *best* we can hope for is something that allows tools to support *both* "Scripts" and "bin". Only after something like that has been in place for an extended period (years) will we have any chance of desupporting environments that use "Scripts".

 

Maybe I’m wrong, and/or it’s not possible on all file systems Python needs to support, in which case *nix systems do support linking, so we could support “Scripts” on all systems.

Just a thought.

I'd strongly encourage anyone who is still interested in pursuing this to at least work on a proof of concept set of changes, rather than just discussing how it would be better than what we're doing at the moment. This has come up enough times now that anyone involved in packaging is pretty much resigned to us not being able to change this, and we're not likely to do anything just because someone says "this is worth doing". However, it's possible we're so resigned to the current situation that we haven't thought of some approach that will work - and if so, the best way to demonstrate that will certainly be to come with working (or at least partly working) code.

If people want to speculate on how a change might be implemented, then please go ahead (this is python-ideas, after all). But without someone willing to write code, that's all it's likely to be, speculation. Sorry.

Paul

[Barry Scott]

On 21 Jul 2022, at 16:42, Christopher Barker <pyth...@gmail.com> wrote:

I still am dumbfounded that this wasn’t platform I dependent in the first place, but you know what essay about hindsight.

However, I’m no Windows expert, but I *think* the modern Windows file system(s?) support something like symlinks. It’s an under-the-hood feature, but maybe it’s possible to add a symlink for bin. 

It has symlinks but only available if you are administrator.

Maybe I’m wrong, and/or it’s not possible on all file systems Python needs to support, in which case *nix systems do support linking, so we could support “Scripts” on all systems.

-1

Barry

Message archived at https://mail.python.org/archives/list/python...@python.org/message/S6DMH3SXK53ZLG5DRFIITVAZFVUNW53Y/

[Barry Scott]

> On 21 Jul 2022, at 00:46, Svein Seldal <sve...@seldal.com> wrote:
>
>
> Using py in the three OS-es (unix-like, Mac and Win) has become very similar in the last years, which is great. Making portable py code has become very easy. However, there is one peculiarity in Win which I find annoying: virtual environments insists on placing executables in the `Scripts` directory, while the other platforms use `bin`. This forces portable scripts and programs on the outside of py to be OS-aware in order to interact with a venv on a Win-system vs other systems.
>
> I would like to proposing the ability to configure virtual environments to use `bin` directory for executable on Windows. Personally I think it would be smart move if it were changed to `bin` as default, making all py platform act consistently. However, I do understand that there might be existing code out there that depend on using `Scripts`.

In my experience I have to write Windows specific scripts and code so dealing with "Scripts" is not an issue.
Yes I would have preferred it to be called bin, but we are where we are.

Barry

>
> Best regards,
> Svein
> _______________________________________________
> Python-ideas mailing list -- python...@python.org
> To unsubscribe send an email to python-id...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at https://mail.python.org/archives/list/python...@python.org/message/5OFYP734PMXBVIXEW444IU4CPUI7KTIB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/EV4OTJN34SCXDJ26QOC4VCQMVNU7ADW5/

[Eryk Sun]

On 7/24/22, Barry Scott <ba...@barrys-emacs.org> wrote:
>
>> On 21 Jul 2022, at 16:42, Christopher Barker <pyth...@gmail.com> wrote:
>
>> However, I’m no Windows expert, but I *think* the modern Windows file
>> system(s?) support something like symlinks. It’s an under-the-hood
>> feature, but maybe it’s possible to add a symlink for bin.
>
> It has symlinks but only available if you are administrator.

Creating symlinks requires the filesystem to support NT reparse
points. That's guaranteed for the system volume, which must be NTFS,
but it's unreliable when development is spread across various
filesystems. This is the main obstacle to relying on a "Scripts" ->
"bin" link.

It's not technically correct to state that creating symlinks requires
administrator access. It requires SeCreateSymbolicLinkPrivilege, or no
privilege at all if developer mode is enabled for the system in
Windows 10+. By default this privilege is granted to just the
administrators group. However, an administrator can grant it to any
user or group. I prefer to grant it to the "Authenticated Users"
group.

If creating a directory symlink isn't allowed, and the filesystem
supports reparse points, then a junction mount point can be created
instead. In Unix terms, this is like using a bind mount instead of a
symlink. In Windows, creating a mount point doesn't require any
privilege or special access. (Registering it with the mount-point
manager requires administrator access, but that's only done for volume
mount points, as created by SetVolumeMountPointW.)

_______________________________________________
Python-ideas mailing list -- python...@python.org
To unsubscribe send an email to python-id...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/

Message archived at https://mail.python.org/archives/list/python...@python.org/message/AS7REUDGBW4QGQPXTNCMGMC4L2ZSUUXI/