Proposal: Remove Linux PGO Testing

[David Anderson]

Hi,

After being faced with the prospect of debugging Yet Another PGO-only Bug, I propose that we stop testing Linux PGO builds. One of these bugs is causing perma-red on Aurora (bug 799295). The developer cost of figuring out how to reproduce, debug, and fix these problems is quite high, and even higher where a specific distro and compiler version might be in play. The cost is not worth the benefit of testing this configuration.

All evidence at Mozilla suggests that PGO is endlessly buggy, and I've heard this has been true historically in compilers. PGO deeply and zealously modifies calling conventions, function bodies, and large callgraphs, causing subtle problems, often topcrashes, that are difficult to reproduce and nearly impossible to debug. The result is that we litter the codebase with untestable deoptimization hints that may or may not randomly become obsolete.

This is depressing on any platform, but at least on Windows, we have a single compiler version to worry about (and, notably, a compiler that has PGO bugs even with a huge amount of resources behind its development). It is possible that PGO could expose real bugs. If so, we'll see the bug one way or another, likely in a context that is more easily reproducible and easier to debug.

So, let's turn off Linux PGO testing, free up releng resources, and save ourselves the inevitable pain and suffering.

-David

[Justin Lebar]

By "turning off Linux PGO testing", you really mean "stop making and
distributing Linux PGO builds," right?

The main reason I'd want Linux PGO is for mobile. On desktop Linux,
most users (I expect) don't run our builds, so it's not a big deal if
they're some percent slower. (Unless distros commonly do PGO builds
of Firefox?) But we're not doing mobile Linux PGO builds (that I know
of), and I don't expect success with desktop PGO is much related to
success with mobile PGO.

-Justin

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[David Anderson]

Yeah, if we're not testing them I guess we don't have to make or distribute them at all.

-David

[Ted Mielczarek]

On Wed, Oct 10, 2012 at 5:57 PM, Justin Lebar <justin...@gmail.com> wrote:
> By "turning off Linux PGO testing", you really mean "stop making and
> distributing Linux PGO builds," right?
>
> The main reason I'd want Linux PGO is for mobile. On desktop Linux,
> most users (I expect) don't run our builds, so it's not a big deal if
> they're some percent slower. (Unless distros commonly do PGO builds
> of Firefox?) But we're not doing mobile Linux PGO builds (that I know
> of), and I don't expect success with desktop PGO is much related to
> success with mobile PGO.

Only inasmuch as the compiler and build machinery continues to work,
but it uses most of the same code as the Windows PGO builds, save the
compiler/linker flags, and those aren't terribly complicated and I
don't expect them to break if we turn the builds off.

I don't really have an opinion about this otherwise, I agree that most
Linux users aren't using our builds anyway.

-Ted

[Mike Hommey]

On Wed, Oct 10, 2012 at 05:57:53PM -0400, Justin Lebar wrote:
> By "turning off Linux PGO testing", you really mean "stop making and
> distributing Linux PGO builds," right?
>
> The main reason I'd want Linux PGO is for mobile. On desktop Linux,
> most users (I expect) don't run our builds, so it's not a big deal if
> they're some percent slower.

Many people have made claims about that at several different occasions.
Can we once and for all come up with actual data on that?

That being said, PGO on Linux is between 5 and 20% improvement on our
various talos tests. That's with the version of gcc we currently use,
which is 4.5. I'd expect 4.7 to do a better job even, especially if we
added lto to the equation (and since we are now building on x86-64
machines, we wouldn't have to worry about memory usage ; link time could
be a problem, though).

Also note that disabling PGO currently also means disabling the
optimizations we do on omni.ja (central directory optimizations and
reordering). This is somehow covered by bug 773171.

Mike

[Tim Taubert]

On 10/10/2012 11:57 PM, Justin Lebar wrote:
> The main reason I'd want Linux PGO is for mobile. On desktop Linux,
> most users (I expect) don't run our builds, so it's not a big deal if

> they're some percent slower. (Unless distros commonly do PGO builds
> of Firefox?) But we're not doing mobile Linux PGO builds (that I know
> of), and I don't expect success with desktop PGO is much related to
> success with mobile PGO.

You may be right for release builds but that doesn't hold true for
Nightly/Aurora/Beta users. I don't think it's a good idea to make those
builds ~20% slower when of course we want and need more testers. Don't
forget that testers on Linux do not only test Linux-only features but
also features we have on every platform.

Nobody likes running debug builds because they're slower so why would
that be different for non-PGO builds?

Also, I'm not sure how this affects Telemetry results. In terms of perf
measurements we'd probably need to completely ignore everything from
non-release builds as the results might differ heavily for some use
cases.

- Tim

[Boris Zbarsky]

On 10/11/12 3:05 AM, Tim Taubert wrote:
> Also, I'm not sure how this affects Telemetry results. In terms of perf
> measurements we'd probably need to completely ignore everything from
> non-release builds as the results might differ heavily for some use
> cases.

I'm not following.

The suggestion, as far as I can tell, is to drop Linux PGO completely.
We woudln't have it in nightly, Aurora, Beta, or releases. Compiling
with PGO on Linux would be an unsupported configuration that we'd
probably advise distros against, because it wouldn't be particularly
well-tested.

So the real question is whether PGO on Linux is worth it in general to
us, again as far as I can tell.

-Boris

[Tim Taubert]

On 10/11/2012 09:32 AM, Boris Zbarsky wrote:
> The suggestion, as far as I can tell, is to drop Linux PGO completely.
> We woudln't have it in nightly, Aurora, Beta, or releases. Compiling
> with PGO on Linux would be an unsupported configuration that we'd
> probably advise distros against, because it wouldn't be particularly
> well-tested.

Yes, if we don't distribute PGO builds at all we'd see a one-time bump
and Telemetry is then a non-issue. I was misguided by the thread's
title. If we turn it off for testing we can't ship it.

- Tim

[David Anderson]

Right, exactly. I am arguing that testing PGO, which is a buggy optimization pass, incurs too much developer cost to justify a "5-20%" talos improvement on select benchmarks. On Linux, which is a very small percentage of our market share, and where distributions make their own builds anyway.

Whether we'd tell distributions that PGO was unsupported: it actually seems difficult to say that it *is* supported, even now. PGO bugs will likely be highly dependent on the environment and compiler version, which are won't be exactly the same anywhere as they are on Windows.

-David

[David Anderson]

Keep in mind that debug builds are probably at least an order of magnitude slower (or a large factor), whereas PGO is a very small factor. (After all, we do not PGO on Mac and it doesn't seem to be a problem.)

-David

[Neil]

Tim Taubert wrote:

>Nobody likes running debug builds because they're slower
>

I always run debug builds. What does that make me? ;-)

--
Warning: May contain traces of nuts.

[Gervase Markham]

On 11/10/12 08:54, David Anderson wrote:
> Keep in mind that debug builds are probably at least an order of
> magnitude slower (or a large factor), whereas PGO is a very small
> factor. (After all, we do not PGO on Mac and it doesn't seem to be a
> problem.)

5-20%, if it were a general slowdown, is _huge_. We have people who work
for months to get speedups of 1 or 2%.

We should find out whether the Linux distros are using PGO. If they are,
it would be unwise for us to stop supporting their release
configuration, and tell them that the new supported configuration is a
lot slower...

Gerv

[David Anderson]

Yeah, if we're not testing them I guess we don't have to make or distribute them at all.

-David

On Wednesday, October 10, 2012 2:58:18 PM UTC-7, Justin Lebar wrote:

[David Anderson]

Keep in mind that debug builds are probably at least an order of magnitude slower (or a large factor), whereas PGO is a very small factor. (After all, we do not PGO on Mac and it doesn't seem to be a problem.)

-David

On Thursday, October 11, 2012 12:05:35 AM UTC-7, Tim Taubert wrote:

[David Anderson]

> 5-20%, if it were a general slowdown, is _huge_. We have people who work
>
> for months to get speedups of 1 or 2%.

Yes, I know, that is pretty much all I do at Mozilla ;) I don't think scattered Talos wins of 5-20% are so valuable and important that we should keep sacrificing developer time to debug problems from having the buggy optimization pass.

> We should find out whether the Linux distros are using PGO. If they are,
>
> it would be unwise for us to stop supporting their release

As I said, unless they're using the exact same compiler version and environment, we're already not supporting their configuration. C++ compilers are twitchy - much more so with PGO.

-David

[Rafael Ávila de Espíndola]

On 10/11/2012 02:33 AM, Mike Hommey wrote:

> On Wed, Oct 10, 2012 at 05:57:53PM -0400, Justin Lebar wrote:
>> By "turning off Linux PGO testing", you really mean "stop making and
>> distributing Linux PGO builds," right?
>>

>> The main reason I'd want Linux PGO is for mobile. On desktop Linux,
>> most users (I expect) don't run our builds, so it's not a big deal if
>> they're some percent slower.
>

> Many people have made claims about that at several different occasions.
> Can we once and for all come up with actual data on that?
>
> That being said, PGO on Linux is between 5 and 20% improvement on our
> various talos tests. That's with the version of gcc we currently use,
> which is 4.5. I'd expect 4.7 to do a better job even, especially if we
> added lto to the equation (and since we are now building on x86-64
> machines, we wouldn't have to worry about memory usage ; link time could
> be a problem, though).
>
> Also note that disabling PGO currently also means disabling the
> optimizations we do on omni.ja (central directory optimizations and
> reordering). This is somehow covered by bug 773171.

I wouldn't be surprised if most of the pgo benefit is because of bad
inline decisions by gcc. If we can narrow the gap by adding
MOZ_ALWAYS_INLINE, then maybe we can drop pgo.

I did a talos run during the last clang update to compare it with the
previous version on OS X, but I now added linux runs to compare gcc 4.5
and clang on linux. The results are interesting (see attached file).
dromaeo_dom shows a 31% improvement on 64 bits for example.

This also suggests another option: using clang on linux too. This would
have the added benefit of using the same compiler for OS X and Linux,
which would remove most of the argument of developers spending time on
linux only issues. I can do a comparison of gcc+pgo and clang if others
are interested.

> Mike

Cheers,
Rafael

[Gary Kwong]

I filed bug 800471 for considering using Clang on Linux.

-Gary

> This also suggests another option: using clang on linux too. This would
> have the added benefit of using the same compiler for OS X and Linux,
> which would remove most of the argument of developers spending time on
> linux only issues. I can do a comparison of gcc+pgo and clang if others
> are interested.
>

> Cheers,
> Rafael
>

[Anthony Jones]

On 11/10/12 19:33, Mike Hommey wrote:
> That being said, PGO on Linux is between 5 and 20% improvement on our
> various talos tests. That's with the version of gcc we currently use,
> which is 4.5. I'd expect 4.7 to do a better job even, especially if we
> added lto to the equation (and since we are now building on x86-64
> machines, we wouldn't have to worry about memory usage ; link time could
> be a problem, though).

Perhaps the problems can be resolved or ameliorated by bumping the
minimum version of GCC that we support for PGO.

[Zack Weinberg]

Link-time optimization is described as an experimental new feature in
the GCC 4.5.0 release notes[1]. The 4.6.0 release notes[2] say that it
has now "stabilized to the point of being usable", and the 4.7.0 release
notes[3] describe it as still further improved both in reliability and
code quality. If we're trying to use the 4.5 LTO, I'm not at all
surprised to hear it's causing more trouble than it's worth.

PGO is not the same thing as LTO, of course, but GCC's PGO was kind of
an unloved stepchild until they got serious about LTO, so that, too, is
likely to be much improved in 4.7.

zw

[1] http://gcc.gnu.org/gcc-4.5/changes.html
[2] http://gcc.gnu.org/gcc-4.6/changes.html
[3] http://gcc.gnu.org/gcc-4.7/changes.html

[Mike Hommey]

On Thu, Oct 11, 2012 at 02:26:33PM -0400, Rafael Ávila de Espíndola wrote:
> On 10/11/2012 02:33 AM, Mike Hommey wrote:
> >On Wed, Oct 10, 2012 at 05:57:53PM -0400, Justin Lebar wrote:
> >>By "turning off Linux PGO testing", you really mean "stop making and
> >>distributing Linux PGO builds," right?
> >>
> >>The main reason I'd want Linux PGO is for mobile. On desktop Linux,
> >>most users (I expect) don't run our builds, so it's not a big deal if
> >>they're some percent slower.
> >
> >Many people have made claims about that at several different occasions.
> >Can we once and for all come up with actual data on that?
> >

> >That being said, PGO on Linux is between 5 and 20% improvement on our
> >various talos tests. That's with the version of gcc we currently use,
> >which is 4.5. I'd expect 4.7 to do a better job even, especially if we
> >added lto to the equation (and since we are now building on x86-64
> >machines, we wouldn't have to worry about memory usage ; link time could
> >be a problem, though).
> >

> >Also note that disabling PGO currently also means disabling the
> >optimizations we do on omni.ja (central directory optimizations and
> >reordering). This is somehow covered by bug 773171.
>
> I wouldn't be surprised if most of the pgo benefit is because of bad
> inline decisions by gcc. If we can narrow the gap by adding
> MOZ_ALWAYS_INLINE, then maybe we can drop pgo.

A non-unsignificant part of the performance improvements PGO gives come
from code reordering to improve branch prediction. Presumably, we can
use NS_LIKELY/NS_UNLIKELY to improve some branches manually.

Mike

[David Anderson]

These Dromaeo improvements will in part be because IonMonkey is not fully JIT'ing these paths yet (a regression we're tracking from Firefox 17).

-David

> a11yr_paint
>
> Fedora 12 - Constantine | (527.681818182, 2.59651983057) -> (413.15, 1.41403960494) 1.2772x better
>
> Fedora 12 x64 - Constantine | (451.454545455, 1.24422992408) -> (349.45, 0.819984718342) 1.2919x better
>
>
>
> dromaeo_css
>
> Fedora 12 - Constantine | (2152.09181818, 12.0144869607) -> (2639.839, 8.95312637755) 1.2266x better
>
> Fedora 12 x64 - Constantine | (2485.29272727, 13.7659449067) -> (2921.445, 17.8096790459) 1.1755x better
>
>
>
> dromaeo_dom
>
> Fedora 12 - Constantine | (142.806181818, 0.744898237637) -> (187.9765, 0.650195498534) 1.3163x better
>
> Fedora 12 x64 - Constantine | (181.576818182, 0.806482009967) -> (232.0514, 1.34925457235) 1.2780x better
>
>
>
> kraken
>
> Fedora 12 - Constantine | (3939.40909091, 71.9270328223) -> (3767.41, 71.3917310166) 1.0457x better
>
> Fedora 12 x64 - Constantine | (3579.29090909, 15.596652341) -> (3446.56, 9.33275577468) 1.0385x better
>
>
>
> num_ctors
>
> CentOS (x86_64) release 5 (Final) | (232.0, None) -> (174.0, None) 1.3333x better
>
> CentOS release 5 (Final) | (232.0, None) -> (174.0, None) 1.3333x better
>
>
>
> sunspider
>
> Fedora 12 - Constantine | (364.172727273, 1.5583412188) -> (345.08, 1.26271089553) 1.0553x better
>
> Fedora 12 x64 - Constantine | (334.363636364, 2.07058055282) -> (321.97, 2.64172734312) 1.0385x better
>
>
>
> tdhtmlr_nochrome_paint
>
> Fedora 12 - Constantine | (417.558727273, 0.728348424302) -> (392.9735, 0.424360042529) 1.0626x better
>
> Fedora 12 x64 - Constantine | (390.3705, 0.633414181048) -> (359.1264, 6.1131889195) 1.0870x better
>
>
>
> tdhtmlr_paint
>
> Fedora 12 - Constantine | (433.884909091, 4.67646883558) -> (402.9236, 4.62643754814) 1.0768x better
>
> Fedora 12 x64 - Constantine | (398.0735, 3.51583823606) -> (368.2883, 1.94210482286) 1.0809x better
>
>
>
> tp5n_main_rss_paint
>
> Fedora 12 - Constantine | (114134400.0, 271577.523722) -> (115129700.0, 280881.107982) 1.0087x worse
>
> Fedora 12 x64 - Constantine | (149395900.0, 563431.271779) -> (148169900.0, 430915.422528) 1.0083x better
>
>
>
> tp5n_paint
>
> Fedora 12 - Constantine | (377.5355, 2.28065749844) -> (316.3663, 2.72038848381) 1.1933x better
>
> Fedora 12 x64 - Constantine | (325.3768, 3.31696113133) -> (278.6602, 3.33314780988) 1.1676x better
>
>
>
> tp5n_xres_paint
>
> Fedora 12 - Constantine | (7249195.0, 114562.779547) -> (7640897.0, 218688.473551) 1.0540x worse
>
>
>
> tpaint
>
> Fedora 12 - Constantine | (214.727272727, 2.47305127989) -> (187.4, 7.74235973322) 1.1458x better
>
> Fedora 12 x64 - Constantine | (203.545454545, 3.0539930444) -> (173.2, 4.81738641223) 1.1752x better
>
>
>
> tresize
>
> Fedora 12 - Constantine | (13.0813363636, 0.276579872514) -> (11.8753, 0.318387813206) 1.1016x better
>
> Fedora 12 x64 - Constantine | (12.0, 0.0) -> (11.0, 0.0) 1.0909x better
>
>
>
> ts_paint
>
> Fedora 12 - Constantine | (706.287090909, 3.15497376791) -> (641.8735, 3.61716922776) 1.1004x better
>
> Fedora 12 x64 - Constantine | (656.445, 1.80961735305) -> (602.1632, 3.31640631973) 1.0901x better
>
>
>
> tscrollr_paint
>
> Fedora 12 x64 - Constantine | (16857.3727273, 4.77056976419) -> (16833.32, 2.39441325698) 1.0014x better
>
>
>
> tspaint_places_generated_max
>
> Fedora 12 - Constantine | (707.2684, 2.72687947138) -> (641.3894, 3.16920410573) 1.1027x better
>
> Fedora 12 x64 - Constantine | (658.7789, 4.1605723528) -> (604.4052, 1.60219463203) 1.0900x better
>
>
>
> tspaint_places_generated_med
>
> Fedora 12 - Constantine | (700.1843, 4.30450967551) -> (637.8155, 4.1260230609) 1.0978x better
>
> Fedora 12 x64 - Constantine | (657.1211, 4.50905085597) -> (603.0895, 2.4221657736) 1.0896x better
>
>
>
> tsvgr_opacity
>
> Fedora 12 - Constantine | (100.727272727, 1.10814017075) -> (86.2, 0.650233384763) 1.1685x better
>
> Fedora 12 x64 - Constantine | (81.45, 0.744935926267) -> (70.9, 0.758605615557) 1.1488x better

[Dave Mandelin]

On Wednesday, October 10, 2012 11:33:31 PM UTC-7, Mike Hommey wrote:
> That being said, PGO on Linux is between 5 and 20% improvement on our
> various talos tests. That's with the version of gcc we currently use,
> which is 4.5. I'd expect 4.7 to do a better job even, especially if we
> added lto to the equation (and since we are now building on x86-64
> machines, we wouldn't have to worry about memory usage ; link time could
> be a problem, though).

Do we have detailed data? I don't know how to interpret '5-20% on various Talos tests' without context. If it's a 10% improvement on something that takes 10ms per pageload, then I don't think it matters. If it cuts 100ms from a whole pageload, then it starts to sound like it matters.

I'd really like to know not only so we can make a good decision now about whether to use PGO, but also so that we can understand why we made the decision later on. PGO bugs cause crashes for our users and are miserable to debug and can feel like a waste of time to developers. I'd like to have a clear, documented understanding of either why we don't need PGO, or why PGO is worth suffering for.

Dave

[Dave Mandelin]

On Wednesday, October 10, 2012 11:33:31 PM UTC-7, Mike Hommey wrote:

> That being said, PGO on Linux is between 5 and 20% improvement on our
> various talos tests. That's with the version of gcc we currently use,
> which is 4.5. I'd expect 4.7 to do a better job even, especially if we
> added lto to the equation (and since we are now building on x86-64
> machines, we wouldn't have to worry about memory usage ; link time could
> be a problem, though).

[Ehsan Akhgari]

On 2012-10-11 4:34 PM, Mike Hommey wrote:

> On Thu, Oct 11, 2012 at 02:26:33PM -0400, Rafael Ávila de Espíndola wrote:
>> On 10/11/2012 02:33 AM, Mike Hommey wrote:
>>> On Wed, Oct 10, 2012 at 05:57:53PM -0400, Justin Lebar wrote:
>>>> By "turning off Linux PGO testing", you really mean "stop making and
>>>> distributing Linux PGO builds," right?
>>>>
>>>> The main reason I'd want Linux PGO is for mobile. On desktop Linux,
>>>> most users (I expect) don't run our builds, so it's not a big deal if
>>>> they're some percent slower.
>>>
>>> Many people have made claims about that at several different occasions.
>>> Can we once and for all come up with actual data on that?
>>>

>>> That being said, PGO on Linux is between 5 and 20% improvement on our
>>> various talos tests. That's with the version of gcc we currently use,
>>> which is 4.5. I'd expect 4.7 to do a better job even, especially if we
>>> added lto to the equation (and since we are now building on x86-64
>>> machines, we wouldn't have to worry about memory usage ; link time could
>>> be a problem, though).
>>>

>>> Also note that disabling PGO currently also means disabling the
>>> optimizations we do on omni.ja (central directory optimizations and
>>> reordering). This is somehow covered by bug 773171.
>>
>> I wouldn't be surprised if most of the pgo benefit is because of bad
>> inline decisions by gcc. If we can narrow the gap by adding
>> MOZ_ALWAYS_INLINE, then maybe we can drop pgo.
>

> A non-unsignificant part of the performance improvements PGO gives come
> from code reordering to improve branch prediction. Presumably, we can
> use NS_LIKELY/NS_UNLIKELY to improve some branches manually.

Don't both of these proposals map to tons of manual work? I'm not
convinced that doing either of those would necessarily be easier than
finding and fixing the PGO bug at hand.

Ehsan

[Bill McCloskey]

On 10/11/2012 03:49 PM, Ehsan Akhgari wrote:
> Don't both of these proposals map to tons of manual work? I'm not
> convinced that doing either of those would necessarily be easier than
> finding and fixing the PGO bug at hand.

The problem is that fixing this one bug might take only a few days, but
that underestimates the true cost of PGO. The fact is that, as long as
we have it turned on, PGO will keep introducing new bugs. Each new line
of code that we write gives PGO an opportunity to miscompile it. This
represents an unbounded amount of work for us. This can be compared to
the benefits PGO provides, which are growing very slowly, if at all
(about 2% per year [1]).

Additionally, I don't think we have a good idea of how many bugs are
caused by GCC PGO. Windows PGO issues often turn into topcrashes. On
Linux, we may not have enough users for these bugs to bubble up far
enough for us to investigate them.

-Bill

[1] http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.29.434

[Brian Smith]

Zack Weinberg wrote:
> Link-time optimization is described as an experimental new feature in
> the GCC 4.5.0 release notes[1]. The 4.6.0 release notes[2] say that
> it has now "stabilized to the point of being usable", and the 4.7.0
> release notes[3] describe it as still further improved both in
> reliability and code quality. If we're trying to use the 4.5 LTO,
> I'm not at all surprised to hear it's causing more trouble than it's
> worth.
>
> PGO is not the same thing as LTO, of course, but GCC's PGO was kind
> of an unloved stepchild until they got serious about LTO, so that,
> too, is likely to be much improved in 4.7.

I think it is important to give Linux users the fastest browser we can give them, because:

1. Linux users tend to be disproportionately influential in the markets we care the most about (web developers, techies)
2. Linux is the foundation of B2G and Firefox for Android, where we *definitely* must deliver the fastest product we can

Now, if it were up to me, I'd try to reproduce this on a build built with the latest stable GCC or latest stable clang, and if that fixes the issue, I'd consider this a big motivation for upgrading to GCC 4.5 to a better compiler, which we need to do anyway for language feature support. Definitely, I don't think we should be adding hacks to our code to work around GCC problems that are already fixed in later releases of GCC. It's better to just make the build fail when the user attempts to use one of those older GCC releases.

Now, if PGO doesn't result in the fastest browser, then of course we should disable PGO.

Or, if there is no better compiler possible, then yes, I think it makes sense to disable PGO temporarily until there is a better compiler available. (And/or, help fix the compiler, either by contributing a patch, or by commissioning somebody else to contribute a patch.)

Cheers,
Brian

[Justin Lebar]

> 2. Linux is the foundation of B2G and Firefox for Android, where we *definitely* must deliver
> the fastest product we can

I totally agree, but it's not clear to me whether continuing to do PGO
on desktop Linux has any effect on our ability to potentially do PGO
on Android/B2G. If we were to stop doing PGO on desktop Linux
tomorrow, would that make it much harder to do PGO on mobile in the
future?

Not to beg the question as to whether we want to do PGO on desktop
Linux. We may want to for the other reasons you suggest...

On Thu, Oct 11, 2012 at 8:49 PM, Brian Smith <bsm...@mozilla.com> wrote:
> Zack Weinberg wrote:
>> Link-time optimization is described as an experimental new feature in
>> the GCC 4.5.0 release notes[1]. The 4.6.0 release notes[2] say that
>> it has now "stabilized to the point of being usable", and the 4.7.0
>> release notes[3] describe it as still further improved both in
>> reliability and code quality. If we're trying to use the 4.5 LTO,
>> I'm not at all surprised to hear it's causing more trouble than it's
>> worth.
>>
>> PGO is not the same thing as LTO, of course, but GCC's PGO was kind
>> of an unloved stepchild until they got serious about LTO, so that,
>> too, is likely to be much improved in 4.7.
>

> I think it is important to give Linux users the fastest browser we can give them, because:
>
> 1. Linux users tend to be disproportionately influential in the markets we care the most about (web developers, techies)
> 2. Linux is the foundation of B2G and Firefox for Android, where we *definitely* must deliver the fastest product we can
>
> Now, if it were up to me, I'd try to reproduce this on a build built with the latest stable GCC or latest stable clang, and if that fixes the issue, I'd consider this a big motivation for upgrading to GCC 4.5 to a better compiler, which we need to do anyway for language feature support. Definitely, I don't think we should be adding hacks to our code to work around GCC problems that are already fixed in later releases of GCC. It's better to just make the build fail when the user attempts to use one of those older GCC releases.
>
> Now, if PGO doesn't result in the fastest browser, then of course we should disable PGO.
>
> Or, if there is no better compiler possible, then yes, I think it makes sense to disable PGO temporarily until there is a better compiler available. (And/or, help fix the compiler, either by contributing a patch, or by commissioning somebody else to contribute a patch.)
>
> Cheers,
> Brian

[David Anderson]

On Thursday, October 11, 2012 5:49:07 PM UTC-7, Brian Smith wrote:
> I think it is important to give Linux users the fastest browser we can give them, because:

It's still unclear to me what our Linux PGO builds mean. Do distributions use them? If not, are they using the exact same compiler version and PGO environment data? If not, then they have a different configuration that we haven't tested.

If distributions are using these builds, or these builds are favored by users over distribution builds, then we'll have to answer Dave and Bill's points re: cost-benefit analysis.

-David

[Mike Hommey]

On Fri, Oct 12, 2012 at 12:28:43AM -0700, David Anderson wrote:
> On Thursday, October 11, 2012 5:49:07 PM UTC-7, Brian Smith wrote:
> > I think it is important to give Linux users the fastest browser we
> > can give them, because:
>
> It's still unclear to me what our Linux PGO builds mean. Do
> distributions use them?

No, distributions build from scratch.

> If not, are they using the exact same compiler version and PGO
> environment data?

No.

> If not, then they have a different configuration that we haven't
> tested.
>
> If distributions are using these builds, or these builds are favored

> by users over distribution builds (...)

We actually don't know. Or if someone knows from actual data rather than
conjecture, they haven't made the information public. I do know that
some people prefer the mozilla builds over distro builds, but I don't
know if they are a lot.

Mike

[David Anderson]

On Thursday, October 11, 2012 5:49:07 PM UTC-7, Brian Smith wrote:

> I think it is important to give Linux users the fastest browser we can give them, because:

It's still unclear to me what our Linux PGO builds mean. Do distributions use them? If not, are they using the exact same compiler version and PGO environment data? If not, then they have a different configuration that we haven't tested.

If distributions are using these builds, or these builds are favored by users over distribution builds, then we'll have to answer Dave and Bill's points re: cost-benefit analysis.

-David

[David Anderson]

These Dromaeo improvements will in part be because IonMonkey is not fully JIT'ing these paths yet (a regression we're tracking from Firefox 17).

-David

[Chris Peterson]

On 10/11/12 7:36 PM, Justin Lebar wrote:
>> 2. Linux is the foundation of B2G and Firefox for Android, where we *definitely* must deliver
>> the fastest product we can
>
> I totally agree, but it's not clear to me whether continuing to do PGO
> on desktop Linux has any effect on our ability to potentially do PGO
> on Android/B2G. If we were to stop doing PGO on desktop Linux
> tomorrow, would that make it much harder to do PGO on mobile in the
> future?

Our testing of PGO Linux builds may be papering over performance
problems that affect non-PGO Linux builds (i.e. B2G, Firefox for
Android, and distro-built Linux builds).


chris

[Brian Smith]

David Anderson wrote:
> It's still unclear to me what our Linux PGO builds mean. Do
> distributions use them? If not, are they using the exact same
> compiler version and PGO environment data? If not, then they have a
> different configuration that we haven't tested.

I agree that we should make sure that we are testing the configuration(s) that users are using, and that Linux distros might be using a different configuration than what we're testing. That is a separate issue of whether the right configuration is PGO or not. If PGO is the fastest and Linux distributors are not distributing PGO builds, then we should help the distros start doing PGO builds.

IMO we should help Linux distributors use the configuration we're testing, and (only) when we we're successful at that should the result be called "Firefox." I don't mean that we should dictate a configuration; rather, we should work together with the Linux packagers so we all agree on the optimal, supported, build configuration, and then incorporate that into the trademark rules.

Cheers,
Brian

[Taras Glek]

Theory:
PGO seems like the only way to balance speed vs size. GCC has the effect
of compiling your hot code with -O3 and cold with -Os.
Sure you can gain similar performance gains by aggressively inlining,
but you are going to pay for that dearly in code size(and subsequently
startup speed and to some degree resident memory usage).

One of the side-effects of being a large program is that code that
frequently runs together has a good likelyhood of being spread around
the binary. This means large apps need larger caches to stay as fast as
smaller apps. PGO offers a way out of this by letting the compiler group
warm code in effect letting your large feature-laden app run as well as
a nimbler more specialized one. Note I had no data to back this up other
than my discussions with GCC devs.

Littering code with manual branchprediction + inlining seems like a
really failure-prone unscalable alternative.

Practice:
In practice GCC PGO only has locality benefits at compilation unit level
which results in the same suboptimal locality when linked. Only when one
throws LTO into the mix is locality handled right. However we have not
checked recently if modern GCC is robust enough for our needs yet. So
the main benefit of PGO atm is faster startup vs -O3 builds.

Since almost nobody uses Mozilla Firefox builds(and no Firefox
disributors do pgo), it may not be that bad to hurt startup for a few
precious Linux users.

Stary-eyed-future:
We need PGO + LTO to generate smallest-possible-fast code on mobile.
Unfortunately I haven't not heard anything reassuring about ARM PGO/LTO
in GCC. It's still likely to be broken as heck.

Taras

PS. Rafael, I'd be very happy to switch to clang if it implemented
guided optimizations. I'd be much more tempted to invest resources in
fixing clang bugs in this area than fixing gcc ones.

[Chris AtLee]

We build & test both PGO and non-PGO on branches like mozilla-inbound
and mozilla-central.

[Alex Keybl]

Bug 799295 [1], the driver for this thread, is still an open issue for FF18 (shipping in 6 weeks). The JS team's recommendation remains to disable PGO on Linux. According to Taras, the major benefits of PGO on Linux are for a "starry-eyed-future". Given

> almost nobody uses Mozilla Firefox builds(and no Firefox disributors do pgo), it may not be that bad to hurt startup for a few precious Linux users.

I'd like to suggest that we move forward with disabling PGO on Linux builds, rather than investigating issues that our users will not see on their non-PGO builds. If you have a differing opinion, please make a recommendation to the JS team for how to resolve bug 799295. The only other recommendation in the bug is to drop dromaeojs talos suite testing on Linux, which I don't have enough context to comment on.

-Alex

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=799295

[Mike Hommey]

On Mon, Nov 12, 2012 at 03:47:32PM -0800, Alex Keybl wrote:
> Bug 799295 [1], the driver for this thread, is still an open issue for
> FF18 (shipping in 6 weeks). The JS team's recommendation remains to
> disable PGO on Linux. According to Taras, the major benefits of PGO on
> Linux are for a "starry-eyed-future". Given
>

> > almost nobody uses Mozilla Firefox builds(and no Firefox disributors
> > do pgo), it may not be that bad to hurt startup for a few precious
> > Linux users.
>

> I'd like to suggest that we move forward with disabling PGO on Linux
> builds, rather than investigating issues that our users will not see
> on their non-PGO builds. If you have a differing opinion, please make
> a recommendation to the JS team for how to resolve bug 799295. The
> only other recommendation in the bug is to drop dromaeojs talos suite
> testing on Linux, which I don't have enough context to comment on.

Another recommendation would be to try with a newer gcc.

Mike

[Dao]

On 13.11.2012 00:47, Alex Keybl wrote:
>> >almost nobody uses Mozilla Firefox builds(and no Firefox disributors do pgo)

We should really get the latter fixed. Disabling PGO for our builds
seems like a step in the wrong direction; the numbers collected in this
thread suggest that it's a major loss.

[Jonathan Kew]

On 12/11/12 15:47, Alex Keybl wrote:
> Bug 799295 [1], the driver for this thread, is still an open issue for FF18 (shipping in 6 weeks). The JS team's recommendation remains to disable PGO on Linux. According to Taras, the major benefits of PGO on Linux are for a "starry-eyed-future". Given
>

>> almost nobody uses Mozilla Firefox builds(and no Firefox disributors do pgo), it may not be that bad to hurt startup for a few precious Linux users.
>

> I'd like to suggest that we move forward with disabling PGO on Linux builds, rather than investigating issues that our users will not see on their non-PGO builds. If you have a differing opinion, please make a recommendation to the JS team for how to resolve bug 799295. The only other recommendation in the bug is to drop dromaeojs talos suite testing on Linux, which I don't have enough context to comment on.
>

> -Alex
>
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=799295
>

According to the bug, this crash occurs on most but not all runs of the
test concerned. That suggests to me that it's perhaps *not* a case of
PGO mis-compiling the code, but rather a latent bug in our code (perhaps
a race condition, for example) that is exposed by changed performance
characteristics of the PGO build.

If that's the case, ignoring the problem won't solve it, but leave it
lurking in the shadows for some future user to trip over when conditions
(e.g. performance) happen to be just right.

JK

[Ehsan Akhgari]

On 2012-11-13 9:56 AM, Jonathan Kew wrote:
> On 12/11/12 15:47, Alex Keybl wrote:
>> Bug 799295 [1], the driver for this thread, is still an open issue for
>> FF18 (shipping in 6 weeks). The JS team's recommendation remains to
>> disable PGO on Linux. According to Taras, the major benefits of PGO on
>> Linux are for a "starry-eyed-future". Given
>>

>>> almost nobody uses Mozilla Firefox builds(and no Firefox disributors
>>> do pgo), it may not be that bad to hurt startup for a few precious
>>> Linux users.
>>

>> I'd like to suggest that we move forward with disabling PGO on Linux
>> builds, rather than investigating issues that our users will not see
>> on their non-PGO builds. If you have a differing opinion, please make
>> a recommendation to the JS team for how to resolve bug 799295. The
>> only other recommendation in the bug is to drop dromaeojs talos suite
>> testing on Linux, which I don't have enough context to comment on.
>>
>> -Alex
>>
>> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=799295
>>
>
> According to the bug, this crash occurs on most but not all runs of the
> test concerned. That suggests to me that it's perhaps *not* a case of
> PGO mis-compiling the code, but rather a latent bug in our code (perhaps
> a race condition, for example) that is exposed by changed performance
> characteristics of the PGO build.
>
> If that's the case, ignoring the problem won't solve it, but leave it
> lurking in the shadows for some future user to trip over when conditions
> (e.g. performance) happen to be just right.

Agreed. Actually, reading the bug closely, there's nothing which says
someone has tried to debug this (it's not even clear if it's
reproducible locally), and it seems like the only evidence that we have
about this being PGO related is that it happens on PGO builds, which is
not enough evidence at all! So disabling PGO because of this bug is a
mistake, unless *at least* someone shows how gcc is miscompiling the
code here.

Cheers,
Ehsan

[Justin Dolske]

On 11/13/12 10:27 AM, Ehsan Akhgari wrote:

> Agreed. Actually, reading the bug closely, there's nothing which says
> someone has tried to debug this (it's not even clear if it's
> reproducible locally), and it seems like the only evidence that we have
> about this being PGO related is that it happens on PGO builds, which is
> not enough evidence at all! So disabling PGO because of this bug is a
> mistake, unless *at least* someone shows how gcc is miscompiling the
> code here.

Well, but aren't you proposing doing more of exactly the problem that
led to this proposal?

David wrote about there being a pattern of bugs that only manifest on
Linux PGO, and that there's a very high cost of having someone figure
out how to reproduce and debug the issue. (Often, apparently, only to
find that the cause is a gcc PGO bug.)

It's quite possible 799295 is a real bug unrelated to PGO, but I take
the proposal as implying that we're more likely to have time to properly
investigate Linux bugs when the noise of caused-by-buggy-PGO issues is
taken out of the mix.

Justin

[Alex Keybl]

If the Snappy initiative (or any other group of Mozillians) has short-term plans to evangelize the perf wins of PGO to Linux distros, I agree that we should leave PGO builds and testing enabled on Linux and further investigate the mysterious crash in bug 799295. Otherwise, our builds/testing should match what our users work with on a daily basis thus preventing an unnecessary time sink for our devs.

Hopefully those involved in the Snappy initiative can help us come to a final decision here.

-Alex

[Ehsan Akhgari]

But the point here is that unless we know for sure that we're dealing
with a compiler bug, disabling Linux PGO builds may just wallpaper over
the problem.

Ehsan

[Justin Dolske]

On 11/13/12 4:34 PM, Ehsan Akhgari wrote:

> But the point here is that unless we know for sure that we're dealing
> with a compiler bug, disabling Linux PGO builds may just wallpaper over
> the problem.

That's quite possible, and I'm sure there are other currently-used ways
to exercise the code that could expose other lurking issues. The
standard can't be perfection. So disabling PGO sounds like it can be a
reasonable tradeoff to have more time to focus on other bugs that
matter. Especially given that distros are not shipping PGO builds, and
so to a certain degree we're just making work for ourselves here.

Alternatively: It's a zero-sum game. We need to be smart about about
where we focus resources, and there seems to be a pretty compelling
claim (IMO, and I've no skin in it :) that Linux PGO is a poor cost/benefit.

Justin

[Ehsan Akhgari]

Let me try to be more clear. Assuming that the assertion that the bug
in question is not caused by the PGO compiler miscompiling, turning off
PGO in order to move on would be the wrong thing to do. If it's only
affecting a single test, then that test should be disabled. Now, we
don't know whether that assumption is correct or not, as far as I can
tell from reading the bug.

Cheers,
Ehsan

[Taras Glek]

On 11/13/2012 3:53 PM, Alex Keybl wrote:
> If the Snappy initiative (or any other group of Mozillians) has short-term plans to evangelize the perf wins of PGO to Linux distros, I agree that we should leave PGO builds and testing enabled on Linux and further investigate the mysterious crash in bug 799295. Otherwise, our builds/testing should match what our users work with on a daily basis thus preventing an unnecessary time sink for our devs.
>
> Hopefully those involved in the Snappy initiative can help us come to a final decision here.

Snappy doesn't cover Linux(only Windows).
The majority of our Linux users also do not use any of our builds or
even use the same compiler as we do. By above logic we should completely
stop doing Linux builds, but that's clear not a good idea :)

This should be fixed. There are ways to disable usage of PGO-information
on a per file basis. Someone could land that to sidestep this crash on
the release channel.

Taras

>
> -Alex
>
> On Nov 13, 2012, at 6:54 AM, Dao <d...@design-noir.de> wrote:
>

[Mike Hommey]

On Tue, Nov 13, 2012 at 05:22:22PM -0800, Ehsan Akhgari wrote:
> On 2012-11-13 5:03 PM, Justin Dolske wrote:

> Let me try to be more clear. Assuming that the assertion that the
> bug in question is not caused by the PGO compiler miscompiling,
> turning off PGO in order to move on would be the wrong thing to do.
> If it's only affecting a single test, then that test should be
> disabled. Now, we don't know whether that assumption is correct or
> not, as far as I can tell from reading the bug.

As a matter of fact, I failed to reproduce a crash locally with a
tinderbox build that is supposed to crash. So either i blatantly failed
at reproducing the conditions that should trigger the crash, or it's a
race condition.

Mike

[Alex Keybl]

> The majority of our Linux users also do not use any of our builds or even use the same compiler as we do. By above logic we should completely stop doing Linux builds, but that's clear not a good idea :)

An attempt at getting closer to building what our Linux users use on a daily basis is not futile. If Linux builds are spread over multiple compilers, then we obviously can't change to a representative compiler. But if a large majority of Linux builds aren't using PGO and we know there's a class of bugs specific to PGO (from experience), it makes complete sense from the quality perspective to match that specific configuration.

Discussions are ongoing as to whether disabling the test is our best path forward here, given engineering opposition to disabling PGO.

-Alex

On Nov 13, 2012, at 10:07 PM, Taras Glek <tg...@mozilla.com> wrote:

> On 11/13/2012 3:53 PM, Alex Keybl wrote:
>> If the Snappy initiative (or any other group of Mozillians) has short-term plans to evangelize the perf wins of PGO to Linux distros, I agree that we should leave PGO builds and testing enabled on Linux and further investigate the mysterious crash in bug 799295. Otherwise, our builds/testing should match what our users work with on a daily basis thus preventing an unnecessary time sink for our devs.
>>
>> Hopefully those involved in the Snappy initiative can help us come to a final decision here.
> Snappy doesn't cover Linux(only Windows).
> The majority of our Linux users also do not use any of our builds or even use the same compiler as we do. By above logic we should completely stop doing Linux builds, but that's clear not a good idea :)
>
> This should be fixed. There are ways to disable usage of PGO-information on a per file basis. Someone could land that to sidestep this crash on the release channel.
>
> Taras
>>
>> -Alex
>>
>> On Nov 13, 2012, at 6:54 AM, Dao <d...@design-noir.de> wrote:
>>

[Justin Dolske]

On 11/13/12 5:22 PM, Ehsan Akhgari wrote:

> Let me try to be more clear. Assuming that the assertion that the bug
> in question is not caused by the PGO compiler miscompiling, turning off
> PGO in order to move on would be the wrong thing to do. If it's only
> affecting a single test, then that test should be disabled. Now, we
> don't know whether that assumption is correct or not, as far as I can
> tell from reading the bug.

I think we're cross-talking about different things. :)

I'm replying in the context of the broad issue raised in the beginning
of this thread, and (I think) you're talking about bug 799295 in the
specific context it being a tracked issue for FF18.

If someone has the time to investigate the bug to get it off the
tracking list and buy a bit more time for a PGO-or-not decision, fine.
I'm not too concerned about what we do for this specific bug. I'm more
interested in helping akeybl get to a final decision on the disabling of
PGO (or not). That's an issue independent of whatever is done with
799295, other that that bug being an immediate forcing function.

Justin

[Dave Mandelin]

On Wednesday, November 14, 2012 10:53:37 AM UTC-8, Alex Keybl wrote:
> Discussions are ongoing as to whether disabling the test is our
> best path forward here, given engineering opposition to disabling
> PGO.

I strongly recommend disabling the test for 32-bit Linux PGO and moving on. Bug 799295 has gotten attention and effort far out of proportion to its likely impact. We all have much better things to do with our time.

Dave

[Dave Mandelin]

On Wednesday, November 14, 2012 10:53:37 AM UTC-8, Alex Keybl wrote:

> Discussions are ongoing as to whether disabling the test is our
> best path forward here, given engineering opposition to disabling
> PGO.