Issue 1109180 in chromium: a python2 precompiled binary should be provided for modern Linux distros

[dpranke via monorail]

Updates:
Cc: pyt...@chromium.org

Comment #9 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c9

cc'ing a few folks that have been involved in this in the past.

I think we need to be careful in thinking about if and how we want to do this, because we've seen in the past breakages come up and it seems even more likely to break things if we mess with python2 on non-Windows platforms.

My rough understanding of the state of the world is something like the following:

- on non-Windows, we always assume there is a system python 2 installed; on Windows we assume there isn't one.
- we never assume there is a python 3 installed.
- on non-Windows, we don't provide a $depot_tools/python3 (bash) wrapper, but we do provide a $depot_tools/python-bin/python3 (bash) wrapper
- on Windows, we provide a $depot_tools/python3.bat and a $depot_tools/python-bin/python3.bat
- on Windows, we use $depot_tools/bootstrap-$PYTHON3_VERSION to hold python2 and python3 initially; the final bootstrapping phase copies those out to $depot_tools.
- on non-Windows we (re-)use the bootstrap directory for a python3 executable. We have a checked-in python3-bin/python3 (bash) wrapper that points to it.

I'm not sure if the bootstrap-$PYTHON3_VERSION directory is (directly) used post-bootstrapping. It is somewhat confusing if we do, because python-$PYTHON3_VERSION/python/bin/python.exe is a python2 executable, not a python3 executable (the python3 executable lives in .../python3/bin/).

I think this bug is potentially looking for a few different things:

1) providing a 'python' binary *somewhere* in depot_tools *that is python 2* on non-Windows.
2) providing a 'python2' binary *somewhere* in depot_tools for non-Windows platforms.
3) providing a 'python' "in-path", i.e. $depot_tools/python`
3a) providing a 'python' "in-path" that is explicitly python2

1) seems reasonable.

2) seems reasonable, but perhaps not all that useful if too many of our scripts assume the python2 thing is called 'python'.

I think we've seen that 3) and 3a) are dangerous, based on our experience with `python3`. Overriding `python2` would presumably be similarly dangerous, and overriding `python` would be presumably much worse on distros that expected `python` to refer to `python3`.

We could adopt a convention like:

- anything that assumes `/usr/bin/env python` must run under both 2 and 3
- anything that assumes `/usr/bin/env python3` must run under 3 (probably don't care if it also runs under 3)
- anything that assumes `/usr/bin/env python2` must run under 2 (probably don't care if it also runs under 2)

I don't know if the broader Python community will guarantee the following things:
- a new (3.8+) Python install will ensure a `python3` exists
- a new (3.8+) Python install will ensure a `python` exists

I assume most distros will do one or the other. I don't know how many will do both, and whether a distro would only provide a `python` if `python2` was not already installed.

PEP-394 seems to explicitly leave some of this open, though it guarantees that a `python` inside a virtualenv will exist. I also don't know how widely PEP-394 is followed.

--
You received this message because:
1. You were specifically CC'd on the issue

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

Reply to this email to add a comment or make updates.

[hferreiro via monorail]

Comment #10 on issue 1109180 by hfer...@igalia.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c10

I opened this bug and created the related CL: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2512906 because Fedora has already moved to python3 only. That also means that python points to python3.

Regarding your points:
1) With the idea of changing it to point to python3 once the transition is complete.
2) Is already available. After running `rm -r bootstrap-3.8.0.chromium.8_bin/ && update_depot_tools`, bootstrap-3.8.0.chromium.8_bin/ contains git/, python/ and python3/, python being Python 2.7.17+chromium.22.
3, 3a) The CL actually add a python and python2 scripts in depot_tools/<bootstrap_dir>/python-bin, alongside python3.

Because the directory needs to be explicitly added to PATH for any of this to work, and python3 already existed there, I thought the CL would be uncontroversial.

[dpranke via monorail]

Comment #11 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c11

My concern with putting the `python` and `python2` shims in the python-bin directory is that if someone has python2 but not python3, then there is no way for them to get python3 without also getting a python2 that overrides their existing one. This might be ok, but it might also be annoying or even cause problems for folks.

It might be safer to put things in a separate directory to allow people to pick and choose which they get.

[dpranke via monorail]

Comment #12 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c12

Of course, creating a separate directory makes life a bit more complicated for people that might need both binaries, but I'm not sure if such people exist on non-Windows platforms ...

[sokcevic via monorail]

Comment #13 on issue 1109180 by sokc...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c13

I agree that user should decide what version should be. I don't think we should default to either python2 or python3.

One option that dpranke@ said is to have two directories. In both, I'd suggest to have python2 and python3 but python will point to one of these.

Alternatives are:
(1) let user do link on their own (maybe README, maybe simple script to generate, e.g. set-default-python). By default, there is no python and python is added to .gitignore
(2) let user choose fist time python is executed (requires interactive shell)

I think (1) could be a good solution especially as we require users to set PATH by hand. As that would be part of instructions, we can call out that set-default-python needs to be executed in order to have `python`.

[hferreiro via monorail]

Comment #14 on issue 1109180 by hfer...@igalia.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c14

Regarding the override, the current build instructions explicitly ask for depot_tools to be added to the end of the path (I guess from the time where the python binaries where provided in the top-level directory). Instructions to add python-bin to the path could be similar, so that there are no overrides.

I think that ideally, the user shouldn't be concerned with this at all and all python scripts should work by having the appropriate shebang, and gn setup to also pick the depot_tools python. Is this even possible?

[dpranke via monorail]

Comment #15 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c15

At the moment, having 'python' be a python3 binary is not going to work in a Chromium build, so in one sense letting people choose isn't all that helpful.

Also, at the moment, getting GN to use the right python binary is harder than I would like. We're working on making that easier as part of flipping the build to use python3 by default. I am hoping that we will be able to flip to python3 by default Real Soon Now; see https://crrev.com/c/2515787. Note that that will explicitly be 'python3' and not 'python'.

We could, in theory, go through and change all the shebang lines to explicitly look for 'python2' rather than 'python', but I would rather people spent the effort simply moving the code to be python3-compatible.

So, I would prefer to not put in a ton of effort on 'python2'-specific things if we can avoid it.

If we find cases where it will be hard and extended amount of work to make code be python3-compatible, we may have to revisit this, though.

[sokcevic via monorail]

Updates:
Cc: bry...@google.com

Comment #16 on issue 1109180 by sokc...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c16

The problem with having depot_tools python being at the end of PATH envvar is this comment by dpranke@

> 2) seems reasonable, but perhaps not all that useful if too many of our scripts assume the python2 thing is called 'python'.

So, if system python points to python3, that we need depot_tools to have higher priority.

bryner@, do you have any suggestions how to move forward with this?

[dpranke via monorail]

Comment #17 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c17

I think we've actually gone back and forth between "depot_tools at the end of the path" and "depot_tools at the beginning" over time for exactly issues like these. For example, right now the Windows instructions say beginning, because we've seen people with non-functional 'python.exe's in the their path already (e.g., Microsoft's Win10 shim binary).

I think the best we can hope for is to give users a way to get a python that is python2 in their PATH, and a way to get a python3 in their PATH, separately, and give them guidance on how to manage the PATH as well. That probably means two different directories.

It might be good to find a table of what 'python' resolves to on different OSes (and versions of OSes) and whether python2 or 3 is available, what version, etc.

[sokcevic via monorail]

Comment #18 on issue 1109180 by sokc...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c18

I think that sounds good. I would just add `python` also adds `python2` as I don't see any downside. If we move forward with this, we should ensure documentation is up to date and users know what they can expect.

[dpranke via monorail]

Comment #19 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c19

That's a fair point. We have shim code now in the tree that looks for a 'python2' on non-Windows platforms in order to guarantee that we get a python2 binary and not a python3 one.

[dpranke via monorail]

Comment #20 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c20

... though ... that would mean there would be no way to get a 'python2' without *also* getting a 'python' that was python2, and ultimately we won't want that.

sigh.

[sokcevic via monorail]

Comment #21 on issue 1109180 by sokc...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c21

Looks like we can't make it work for all. If we provide python2 and python3, in separate directories, we can let user decide which ones they want to import (none, just one, or both). Then, we can let user know how to setup python to point to python2 too (that should be part of documentation and not in codebase). This can be system wide, or session based (again, user can decide what they want). I think that solves original problem of providing python2 binary.

[bryner via monorail]

Comment #22 on issue 1109180 by bry...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c22

I'm not sure at this point what our assumptions are about the binary named just 'python'. I'll have to dig a bit and follow up.

[bugdroid via monorail]

Comment #23 on issue 1109180 by bugdroid: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c23

The following revision refers to this bug:
https://chromium.googlesource.com/chromium/tools/depot_tools/+/26cd974abfad06add50e162a3ebf03e6bc4f793e

commit 26cd974abfad06add50e162a3ebf03e6bc4f793e
Author: Henrique Ferreiro <hfer...@igalia.com>
Date: Wed Nov 11 21:35:58 2020

Add a python2 shim under python2-bin/

Similarly to how python3 is available at python-bin/, this will allow
developers to opt-in to a depot_tools provided python2.

Bug: 1109180
Change-Id: I872eaa1289bb477da73373e99f5e6188a87d39dd
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/2512906
Reviewed-by: Josip Sokcevic <sokc...@google.com>
Reviewed-by: Dirk Pranke <dpr...@google.com>
Commit-Queue: Henrique Ferreiro <hfer...@igalia.com>

[add] https://crrev.com/26cd974abfad06add50e162a3ebf03e6bc4f793e/python2-bin/python2

[bugdroid via monorail]

Comment #24 on issue 1109180 by bugdroid: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c24

The following revision refers to this bug:

https://chromium.googlesource.com/chromium/src.git/+/62f7b02a6a9ade9ab3f2094a843e238728be0926

commit 62f7b02a6a9ade9ab3f2094a843e238728be0926
Author: chromium-autoroll <chromium...@skia-public.iam.gserviceaccount.com>
Date: Thu Nov 12 01:43:33 2020

Roll Depot Tools from 8630bb119f96 to 26cd974abfad (3 revisions)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/8630bb119f96..26cd974abfad

2020-11-11 hfer...@igalia.com Add a python2 shim under python2-bin/
2020-11-11 mat...@microsoft.com Retry CIPD client download on Linux / macOS
2020-11-11 vad...@chromium.org [cipd] Update cipd and vpython to latest versions.

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/depot-tools-chromium-autoroll
Please CC a...@google.com,apo...@google.com,ehmal...@google.com,sokc...@google.com on the revert to ensure that a human
is aware of the problem.

To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/master/autoroll/README.md

Bug: chromium:1109180,chromium:1124112,chromium:1145741
Tbr: a...@google.com,apo...@google.com,ehmal...@google.com,sokc...@google.com
Change-Id: I7be92c6400fac4d3c16662308f000e4b89cc256a
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2533147
Reviewed-by: chromium-autoroll <chromium...@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium...@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#826589}

[modify] https://crrev.com/62f7b02a6a9ade9ab3f2094a843e238728be0926/DEPS

[hferreiro via monorail]

Updates:
Status: Fixed

Comment #25 on issue 1109180 by hfer...@igalia.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c25

(No comment was entered for this change.)

[dpranke via monorail]

Comment #26 on issue 1109180 by dpr...@google.com: a python2 precompiled binary should be provided for modern Linux distros
https://bugs.chromium.org/p/chromium/issues/detail?id=1109180#c26

As a follow-on to this bug, I've written up a further analysis of options in bug 1158013, and still feel pretty good about what we ended up with here.