Intent to require Python 3 to build Firefox 59 and later

[Gregory Szorc]

For reasons outlined at
https://bugzilla.mozilla.org/show_bug.cgi?id=1388447#c7, we would like to
make Python 3 a requirement to build Firefox sometime in the Firefox 59
development cycle. (Firefox 59 will be an ESR release.)

The requirement will likely be Python 3.5+. Although I would love to make
that 3.6 if possible so we can fully harness modern features and
performance.

I would love to hear feedback - positive or negative - from downstream
packagers and users of various operating systems and distributions about
this proposal.

Please send comments to dev-b...@lists.mozilla.org or leave them on bug
1388447.

[Xidorn Quan]

I'm happy hearing this. I would be interested on whether we are going to
drop Python 2 at some point, or are we stuck with that forever?

Also I'm curious what modern features are the team looking forward to?

- Xidorn

[Gregory Szorc]

On Fri, Nov 10, 2017 at 4:22 PM, Xidorn Quan <m...@upsuper.org> wrote:

> I'm happy hearing this. I would be interested on whether we are going to
> drop Python 2 at some point, or are we stuck with that forever?
>

Let's put it this way: we still have a few uses of Perl in the build
system. There have been no meaningful changes to that Perl code in years.

We'll likely be using Python 2 for years ahead because there isn't a
compelling reason to port code that isn't being actively updated.

>
> Also I'm curious what modern features are the team looking forward to?
>

asyncio is huge for performance, especially for I/O bound workloads (both
local and network). We have a few of these in critical paths in the build
system.

Python 3.6 is faster than 2.7 pretty much across the board in everything
except for process startup overhead.

Python 3 has sensible behavior with regards to not coercing Unicode and
bytes types, which means people not using English in their operation system
won't experience as many build failures due to us failing to handle Unicode
in the build system.

Type annotations make it easier to hack on large code bases and make IDEs
more useful.

If you do perf hacking in Python, Python 3 also has a slew of new
facilities for memory and CPU profiling.

There's also a ton of random package additions and improvements in the
standard library. And of course tons of small bug fixes.

>
> On Sat, Nov 11, 2017, at 10:27 AM, Gregory Szorc wrote:
> > For reasons outlined at
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1388447#c7, we would like
> to
> > make Python 3 a requirement to build Firefox sometime in the Firefox 59
> > development cycle. (Firefox 59 will be an ESR release.)
> >
> > The requirement will likely be Python 3.5+. Although I would love to make
> > that 3.6 if possible so we can fully harness modern features and
> > performance.
> >
> > I would love to hear feedback - positive or negative - from downstream
> > packagers and users of various operating systems and distributions about
> > this proposal.
> >
> > Please send comments to dev-b...@lists.mozilla.org or leave them on
> bug
> > 1388447.

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[David Burns]

My only concern about this is how local developer environments are going to
be when it comes to testing. While I am sympathetic to moving to python 3
we need to make sure that all the test harnesses have been moved over and
this is something that needs a bit of coordination. Luckily a lot of the
mozbase stuff is already moving to python 3 support but that still means we
need to have web servers and the actual test runners moved over too.

David

On 10 November 2017 at 23:27, Gregory Szorc <g...@mozilla.com> wrote:

> For reasons outlined at https://bugzilla.mozilla.org/
> show_bug.cgi?id=1388447#c7, we would like to make Python 3 a requirement
> to build Firefox sometime in the Firefox 59 development cycle. (Firefox 59
> will be an ESR release.)
>
> The requirement will likely be Python 3.5+. Although I would love to make
> that 3.6 if possible so we can fully harness modern features and
> performance.
>
> I would love to hear feedback - positive or negative - from downstream
> packagers and users of various operating systems and distributions about
> this proposal.
>
> Please send comments to dev-b...@lists.mozilla.org or leave them on bug
> 1388447.
>
> _______________________________________________

> dev-builds mailing list
> dev-b...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-builds
>
>

[Mike Hommey]

On Fri, Nov 10, 2017 at 05:42:01PM -0800, Gregory Szorc wrote:
> On Fri, Nov 10, 2017 at 4:22 PM, Xidorn Quan <m...@upsuper.org> wrote:
>
> > I'm happy hearing this. I would be interested on whether we are going to
> > drop Python 2 at some point, or are we stuck with that forever?
> >
>
> Let's put it this way: we still have a few uses of Perl in the build
> system. There have been no meaningful changes to that Perl code in years.
>
> We'll likely be using Python 2 for years ahead because there isn't a
> compelling reason to port code that isn't being actively updated.
>
>
> >
> > Also I'm curious what modern features are the team looking forward to?
> >
>
> asyncio is huge for performance, especially for I/O bound workloads (both
> local and network). We have a few of these in critical paths in the build
> system.
>
> Python 3.6 is faster than 2.7 pretty much across the board in everything
> except for process startup overhead.
>
> Python 3 has sensible behavior with regards to not coercing Unicode and
> bytes types, which means people not using English in their operation system
> won't experience as many build failures due to us failing to handle Unicode
> in the build system.

In fairness, those problems wouldn't be magically be fixed by using
python 3. We'd still have to handle the difference between bytes and
unicode somehow, and the reason there have been these problems is that
we're failing to do that, and python 3 alone wouldn't solve that.

Mike

[Gregory Szorc]

Python 3 refuses to perform automatic type coercion between bytes and
unicode. Unlike Python 2, if you mix types without an explicit encode() or
decode(), Python 3 complains loudly for *all* values, not just values that
can't be converted losslessly using the current default encoding (likely
ascii). In my experience, this catches a lot of "this function isn't
handling types consistently" bugs before they occur. Yes, you can still
have bugs due to e.g. improper encoding/decoding. But at least you can more
easily find and audit where encoding/decoding occurs because automatic type
coercion doesn't occur.

[Andreas Tolfsen]

Also sprach Gregory Szorc:

> The requirement will likely be Python 3.5+. Although I would
> love to make that 3.6 if possible so we can fully harness modern
> features and performance.

Hold your horses with 3.6. Only Python 3.5.3 is at the time of
writing available in the most common Linux distributions. I
wouldn’t discount that 3.6 will be generally available in the
short- to medium term.

[Andrew Halberstadt]

On Fri, Nov 10, 2017 at 9:44 PM David Burns <dbu...@mozilla.com> wrote:

> My only concern about this is how local developer environments are going
> to be when it comes to testing. While I am sympathetic to moving to python
> 3 we need to make sure that all the test harnesses have been moved over and
> this is something that needs a bit of coordination. Luckily a lot of the
> mozbase stuff is already moving to python 3 support but that still means we
> need to have web servers and the actual test runners moved over too.
>
> David
>

For libraries like mozbase, I think the plan will be to support both 2 and
3 at
the same time. There are libraries (like 'six') that make this possible.
I'd bet
there are even parts of the build system that will still need to support
both at
the same time.

With that in mind, I don't think python 3 support for test harnesses needs
to
block the build system.

Andrew

On 10 November 2017 at 23:27, Gregory Szorc <g...@mozilla.com> wrote:
>
>> For reasons outlined at

>> https://bugzilla.mozilla.org/show_bug.cgi?id=1388447#c7, we would like

>> to make Python 3 a requirement to build Firefox sometime in the Firefox 59
>> development cycle. (Firefox 59 will be an ESR release.)
>>

>> The requirement will likely be Python 3.5+. Although I would love to make
>> that 3.6 if possible so we can fully harness modern features and
>> performance.
>>

[David Burns]

I am not saying it should but if we have a requirement for python 3, we are
also going to have a requirement for py2 to both be available for local
development.

David

On 11 November 2017 at 14:10, Andrew Halberstadt <ahalbe...@mozilla.com>
wrote:

[Jeff Gilbert]

As long as we follow PEP 394, I'm super excited. (including on our
mozilla-build windows system, which counts as 'unix-like')

[Rok Garbas]

It would make my heart sing if all python 3 code we write would have
defined types (using mypy in in ./mach lint). It would definitely remove
the feeling of "walking blindfolded" when doing any change with the python
tooling. At least I hope providing types would be allowed if not required.

-- Rok

--
-- Rok Garbas

[Gregory Szorc]

On Fri, Nov 10, 2017 at 3:27 PM, Gregory Szorc <g...@mozilla.com> wrote:

> For reasons outlined at https://bugzilla.mozilla.org/s
> how_bug.cgi?id=1388447#c7, we would like to make Python 3 a requirement
> to build Firefox sometime in the Firefox 59 development cycle. (Firefox 59
> will be an ESR release.)
>
> The requirement will likely be Python 3.5+. Although I would love to make
> that 3.6 if possible so we can fully harness modern features and
> performance.
>
> I would love to hear feedback - positive or negative - from downstream
> packagers and users of various operating systems and distributions about
> this proposal.
>
> Please send comments to dev-b...@lists.mozilla.org or leave them on bug
> 1388447.
>

We decided to wait until after the ESR release to establish the Python 3.5+
requirement. Then ESR got bumped to 60.

Nightly is now 61 and free of ESR. And I just submitted patches for review
that establish the Python 3.5+ requirement.

So this is a courtesy heads up that builds will soon require Python 3.5+.
`configure` today has a "checking for Python 3" line that will tell you if
things will "just work" once this lands.