Python 3 is coming (finally!) for Chromium src

[Dirk Pranke]

Hi all,

We're finally at a point where I see a fairly clear path to getting Chromium migrated over to Python3 (yes, we're very late in doing so), so I've written up a doc that outlines a rough plan for doing so (it also includes a FAQ!).

Executive summary: there are some bits of work left to do, but most of the necessary infra work needed to move things to python3 has been done, so I think it's safe to start moving things to be 3-compatible in earnest. But, We do not have an end date or a real project for this yet, and we need help.

Notably, assuming no one strongly objects, I'd like to land this CL ASAP which will flip the bulk of the build's python actions to be run under 3 by default (though some stuff still needs to be and will be run under Python 2), so that we can start preventing new Python2-isms from landing.

Once we do that, I'd like to ask for help from folks to migrate their code over as needed so that we can get this over the finish line. I will be filing bugs to track the work that is needed and farming them out as I can best guess, but please feel free to ask how you can help as well.

This does not mean that we can drop Python 2 just yet. Python 2 is still required for large chunks of stuff, and my current thinking is that in order to avoid having everyone have to worry about which stuff can use new features and which can't, I'd like for us to try and migrate to code that is compatible with both 2 and 3 for now. It's been my experience with our codebase (and other 2->3 migrations) that this is fairly doable. If we find things where it really isn't, though, we can certainly revisit this aspect of the plan.

Also, my current thinking and the thinking of the infra team is that we can target Python 3.8 as the version of Python 3. 3.9 is still a little bleeding-edge, and it's starting to drop more things that were kept around 2-compatibility, so I suspect it'd be harder to write code that is 2- and 3.9-compatible.

If anyone has a build environment where they have Python3 but only some version that is older than 3.8 and upgrading to 3.8 will be an issue, please let me know ASAP!

Please let me know if you have any questions, either by replying to this thread, commenting on the doc, or asking on python@. Once any dust from this thread settles, I'll also move that doc into //docs in the repo and update the Python style guide and other docs as needed.

Thanks,

-- Dirk

[Dirk Pranke]

Update: I've tentatively landed https://crrev.com/c/2515787, which will cause GN and Ninja to use Python3 for some things. I expect this'll break bots and need to be reverted, so this landing is mostly to suss out those breakages.

This will affect you as a developer as well, because you'll need to have `python3` in your path. If you don't currently, you can either install a copy yourself (I recommend 3.8+), or add the `python-bin` subdirectory of depot_tools to your path.

If you have any questions or issues with the above, please reply back and/or send a note to python@, as noted before.

Thanks!

-- Dirk

[Scott Violet]

Did you consider putting this in the depot_tools directory (not a subdirectory). So that all of us don't have to update our path?

  -Scott

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEoffTDgwRNwqhvMAxFEMVGMaB5i7BQ2AFtSEooZ%3DHry2WY-LA%40mail.gmail.com.

[Mike Frysinger]

that was done at some point and broke a lot of people.  we can't inject standard programs (like python3) into $PATH on users.

-mike

You received this message because you are subscribed to the Google Groups "python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python+un...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/python/CAKARY_myyvk3-mN9A1QZqgjZiqDfm2hqEw6O64a%2BHxtZ7zrOjg%40mail.gmail.com.

[Scott Violet]

Seems like we'll break people either way. The question is which is the more common. IMO I like adding a single entry to my path and having everything work.

  -Scott

[Mike Frysinger]

breaking random system tools unrelated to Chromium sounds way worse, especially when devs don't get a choice -- we require depot_tools to be in $PATH to build Chromium.

asking the significant minority of people who build Chromium on a system that lacks standard `python3` in their system to update their $PATH sounds like the better trade-off.

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/python/CAKARY_naJpzdMTG_bMLNQk8t2ZWcsNauw1jL-XfNu9a0bn48PQ%40mail.gmail.com.

[Scott Violet]

What could possibly be more important than devs?!? :)

I'll take your word for it as you have more experience than I in this area.

  -Scott

[Joe Mason]

I thought we had "vpython" for builds to ensure that we could build with the python from depot_tools no matter what version of python is on the system. Why isn't the build using vpython3, and then devs don't need to be bothered about the path? (Or am I misunderstanding how vpython works?)

Using the system python3 seems like it would cause version confusion. I remember the bad old days of having multiple python2's installed and having to be sure the build picked the depot_tools one instead of the random system one.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/python/CAKARY_n9ojQ6YZjzxxVOrcmnUedo_hkEuT7S-x%2B%3DaD2HfBj3TA%40mail.gmail.com.

[Dirk Pranke]

My reasoning matches Mike's; I think we need to be careful when overriding commonly installed tools like `python`. If we put the python3 binary in the same place as everything else, you'd have no option but to use our python binary, and that might not work very well.

-- Dirk

[Xiyuan Xia]

I hit errors like when building chromeos-chrome. Have we fully tested that all scripts should work?

FAILED: gen/chromeos/components/file_manager/resources/main.html gen/chromeos/components/file_manager/resources/main.html.stamp
python3 ../../chromeos/components/file_manager/resources/gen_main_html.py --source ../../ui/file_manager/file_manager/main.html --target gen/chromeos/components/file_manager/resources/main.html
  File "../../chromeos/components/file_manager/resources/gen_main_html.py", line 27                                                                                                                  
    print line + '    ' + _SWA
          ^                                                                                      
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(line + '    ' + _SWA)?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAH%3DT95SBmTffEvf7PxNYrodcXXXGjE9D80tP8Z1_fTRQA8k_Lg%40mail.gmail.com.

[Dirk Pranke]

You are correct that having to juggle python versions is annoying and to some extent that's the point of vpython.

At the moment we believe that there are issues with using vpython instead of python (namely that it slows down the build too much to be acceptable).

But, this belief isn't that well vetted and so we're still looking into it (http://crbug.com/948846). It may be that we'll switch to vpython at some point.

Separately, we have tripped over a few breakages already since I landed the CL, so we're reverting until we can do a pass over things and fix them.

-- Dirk

[Dirk Pranke]

On Fri, Dec 11, 2020 at 2:18 PM Xiyuan Xia <xiy...@chromium.org> wrote:

I hit errors like when building chromeos-chrome. Have we fully tested that all scripts should work?

Our messages just crossed in the ether, so this breakage is known and there's a fix posted for it. However, I'm also reverting the CL for now because I don't want things to be arbitrarily broken over the weekend when I might not be around to fix things.

We did do a bunch of testing and of course the change went through the CQ like everything else, but it's simply not possible to know about every possible build permutation and code path ahead of time, so these sorts of issues are to be expected. I don't know offhand why this particular code path wasn't hit before; I need to look and see if it's only in something not in the CQ.

I should've also noted that you can work around the change if necessary by running `gn gen --script-executable=python` to force GN to use Python 2.

-- Dirk

[Mike Frysinger]

you can always ask to see the raw data ;)

https://crbug.com/1017812 is the bug from when we were putting python3 into depot_tools (and thus $PATH).  i vaguely recall there being threads on chromium-dev too, but i'm going to be too lazy to dig them up if that bug is good enough to convince you.

-mike

[Mike Frysinger]

is it useful to file bugs about failures from python->python3 ?  or should we (CrOS) just keep waiting on the slow slog of bugfixes ?  there's a lot of KI, so maybe working through those first is more useful than piling more (possibly duplicate) things on top.

the most immediate one that jumps out is depot_tools' handling of gsutil.  it still invokes old versions that only support python2.

-mike

--
You received this message because you are subscribed to the Google Groups "python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python+un...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/python/CAEoffTDuaZvsgUBbAHMKoYPqKVL3tC9BTain41oR5x1-HFd5Mg%40mail.gmail.com.

[Dirk Pranke]

We are still a good distance away from a world where everything is invoking 'python3' instead of 'python', so I expect there are many places where things are expecting to be invoked as 'python' or things will be invoking 'python' to run something.

That said, it's always useful to file bugs.

-- Dirk