Quantum Flow Engineering Newsletter #14

[Ehsan Akhgari]

Hi everyone,

We have about 13 more weeks before the train of Firefox 57 leaves the
station. Next week many of you will be at the upcoming work week, so I
thought it may be a good time to have some retrospection over our progress
so far, just so that you can get a good sense of how to extrapolate when
you are planning things next week.

One difficulty with the Quantum Flow project is that since it touches many
different areas of the browser, it doesn't lend itself very easily to
drawing nice charts for it. :-) It is hard to find one metric that all of
this work fits inside, and that's OK. My goal this week is to highlight
what we can achieve with focus in a limited amount of time, so I'll bring a
couple of examples.

This is a snapshot of our burndown chart1. We currently have 182 closed
bugs and 136 open bugs. That's great progress, and I'd like to thank
everyone who helped with all aspects of this!

<https://people-mozilla.org/~cpeterson/bb/burndown.html?whiteboard=[qf:p1]&Since=2017-01-01>

But to speak of a more direct measurement of performance, let's look at our
progress on Speedometer V2
<https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>.
Today, I measured our progress so far on this benchmark by comparing
Firefox 53, 54, 55.0b3 (latest beta as of this writing) and the latest
Nightly, all x64 builds, on the reference hardware
<https://www.amazon.com/dp/B01K1IO3QW>. This is the result (numbers are
the reported benchmark score, higher is better):

[image: Speedometer improvements]

There are also many other top level performance related projects that are
ongoing and approaching final stages. I'm really excited to see what the
next few months are going to uncover for Firefox performance.

One bit of administrative note, as next week most people are able to get
updates from each other face to face, I won't send out the newsletter. Now
let's finish with this week's list of acknowledgements to those who helped
make Firefox faster during the past week, hopefully I'm not forgetting any
names!

- Jan de Mooij optimized object property enumeration
<https://bugzilla.mozilla.org/show_bug.cgi?id=1373615> with great results
<https://treeherder.mozilla.org/perf.html#/alerts?id=7394>. He finished
his ongoing optimization effort on adding/defining properties
<https://bugzilla.mozilla.org/show_bug.cgi?id=1372182>. He also reordered
the checks in ValueToId() and ValueToIdPure() to cover the more common
cases first <https://bugzilla.mozilla.org/show_bug.cgi?id=1370210>. He
then moved the object/string pre-barrier null check to JIT code
<https://bugzilla.mozilla.org/show_bug.cgi?id=1373290>. This speeds up
initializing unboxed objects.
- Makoto Kato improved the performance of HTMLInputElement.value setters
by avoiding doing some unnecessary work
<https://bugzilla.mozilla.org/show_bug.cgi?id=1368888>.
- Alessio Placitelli deferred querying whether we're the default browser
to until after session restore completes
<https://bugzilla.mozilla.org/show_bug.cgi?id=1367029> in order to avoid
potential startup slowdown issues.
- Markus Stange made arrow
<https://bugzilla.mozilla.org/show_bug.cgi?id=1370034> panel animations
<https://bugzilla.mozilla.org/show_bug.cgi?id=1291457> much more
efficient on macOS!
- Doug Thayer made it so that we shrink the Places SQLite database off
of the main thread
<https://docs.google.com/document/d/1FezR5jSInhdDJ7D12Ynrbe3JxGnAvQxwU30O1vFUYxY/edit>,
avoiding some jank when hitting memory pressure.
- Jonathan Kew switched to using a faster API for retrieving font glyph
advances instead of all glyph metrics on DirectWrite
<https://bugzilla.mozilla.org/show_bug.cgi?id=1365776>.
- Mark Banner converted the bookmarks backup code to use the new
asynchronous bookmarks API
<https://bugzilla.mozilla.org/show_bug.cgi?id=1095426>.
- Matthew Noorenberghe made the context menu code lazily import
LoginManagerContextMenu.jsm and InlineSpellChecker.jsm
<https://bugzilla.mozilla.org/show_bug.cgi?id=1374686> at startup.
- Florian Quèze avoided attaching an XBL binding to hidden <xul:label>
elements <https://bugzilla.mozilla.org/show_bug.cgi?id=1374257> to
improve startup performance.
- Boris Zbarsky avoided querying the preferences service to determine
whether interface enablement conditions are met
<https://bugzilla.mozilla.org/show_bug.cgi?id=1374119>.
- Jonathan Watt optimized calls to nsUXThemeData::InitTitlebarInfo()
<https://bugzilla.mozilla.org/show_bug.cgi?id=1369508>, improving the
performance of opening a window on Windows.
- Nicholas B. Pierron enabled backtracking to a normal call when
inlining fails in IonMonkey
<https://bugzilla.mozilla.org/show_bug.cgi?id=1373323>.
- Mats Palmgren continued his
<https://bugzilla.mozilla.org/show_bug.cgi?id=1374089> battle
<https://bugzilla.mozilla.org/show_bug.cgi?id=1374126> on unnecessary
hashtable lookups. He added a couple of new APIs for adding and
removing entries on nsTHashtable which return values indicating whether
they changed the hashtable
<https://bugzilla.mozilla.org/show_bug.cgi?id=1371928>, which allow
rewriting patterns where code currently does two hashtable lookups, first
to check whether an entry exists in the hashtable and then to add/remove it
(which incurs a second lookup on the same entry). He then followed up
with <https://bugzilla.mozilla.org/show_bug.cgi?id=1371925> various
<https://bugzilla.mozilla.org/show_bug.cgi?id=1371956> fixes
<https://bugzilla.mozilla.org/show_bug.cgi?id=1371958> to
<https://bugzilla.mozilla.org/show_bug.cgi?id=1371960> reduce
<https://bugzilla.mozilla.org/show_bug.cgi?id=1372009> duplicate
<https://bugzilla.mozilla.org/show_bug.cgi?id=1372029> hashtable
<https://bugzilla.mozilla.org/show_bug.cgi?id=1372031> lookups
<https://bugzilla.mozilla.org/show_bug.cgi?id=1374125>. He also
devirtualized
nsTableCellFrame::GetRowSpan/GetColSpan()
<https://bugzilla.mozilla.org/show_bug.cgi?id=1373095>.
- Andreas Farre exposed the idle dispatch API with timeout to chrome JS
<https://bugzilla.mozilla.org/show_bug.cgi?id=1368072>.s This is the
final remaining bit of API that brings chrome JS to parity with C++ in
terms of its ability to use the new idle dispatch facilities.
- Cervantes Yu moved our in-process crash reporter to use
off-main-thread IO for generating the crash minidump
<https://bugzilla.mozilla.org/show_bug.cgi?id=1360308>. This means that
our UI will now remain responsive if a tab crashes while we prepare the
corresponding crash report.
- Makoto Kato made UpdateOverlayTextVisibility() not get called twice by
the HTMLInputElement.value setter when the element has focus
<https://bugzilla.mozilla.org/show_bug.cgi?id=1360162>.
- Edgar Chen made document.hidden be set to true when the browser window
is fully covered by another non-translucent application
<https://bugzilla.mozilla.org/show_bug.cgi?id=1236512>. This has some
performance benefits such as treating such pages as if they were in a
background tab and throttling down timeouts/refresh driver for them.
- Jean-Yves Avenard added support for texture recycling to the ffmpeg
media backend on Windows
<https://bugzilla.mozilla.org/show_bug.cgi?id=1223270>.
- Kirk Steuber made us clone attributes rather than reparse them
<https://bugzilla.mozilla.org/show_bug.cgi?id=1357645> when possible
when cloning a node.
- Olli Pettay made nsTextEditorState::UpdateOverlayTextVisibility()
avoid accessing the preferences service every time
<https://bugzilla.mozilla.org/show_bug.cgi?id=1374117>.
- Doug Thayer moved storage cache shrinking to happen off the main thread
<https://bugzilla.mozilla.org/show_bug.cgi?id=1166166>.
- Jon Coppeard changed Zone::isGCMarking() to avoid a TLS lookup
<https://bugzilla.mozilla.org/show_bug.cgi?id=1373214>.
- Yoshi Huang moved nsImageLoadingContent to the Necko AsyncOpen2 API
<https://bugzilla.mozilla.org/show_bug.cgi?id=1267075> which, among
security benefits, also reduced the cost of loading images by removing some
security checks which happen centrally at the networking layer now.
- Lee Salzman optimized box blur surfaces for destination draw targets
<https://bugzilla.mozilla.org/show_bug.cgi?id=1365794>. This helps with
the performance of YouTube videos with text overlays using box shadows,
which is common.
- Dão Gottwald avoided a layout flush in order to determine whether
scrollbox scroll buttons should be enabled/disabled
<https://bugzilla.mozilla.org/show_bug.cgi?id=1368208>.
- Ted Campbell improved JIT
<https://bugzilla.mozilla.org/show_bug.cgi?id=1169746> support
<https://bugzilla.mozilla.org/show_bug.cgi?id=1169743> for ES6 classes.
Made nice improvements to the ARES-6 benchmark!
[image: Drop in ARES-6 benchmark numbers]

[1] (The number of bug fixes is a weird metric to use for performance
improvements, since we use bugs as a unit of work, and the performance
impact of each bug can be vastly different. But I have tried to describe
the details of these bugs for the most part before so the detailed
information is at least available.)
Cheers,
--
Ehsan

[Chris Peterson]

On 6/23/17 12:17 AM, Ehsan Akhgari wrote:
>
> But to speak of a more direct measurement of performance, let's look
> at our progress on Speedometer V2
> <https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>.
> Today, I measured our progress so far on this benchmark by comparing
> Firefox 53, 54, 55.0b3 (latest beta as of this writing) and the latest
> Nightly, all x64 builds, on the reference hardware
> <https://www.amazon.com/dp/B01K1IO3QW>. This is the result (numbers
> are the reported benchmark score, higher is better):
>

> Speedometer improvements
>

How do these Speedometer V2 scores map to the results on AWFY? AWFY
shows many Speedometer sub-tests, but no score in the range of 70.21.
AWFY machine #36 is the reference hardware.

https://arewefastyet.com/#machine=36&view=breakdown&suite=speedometer-misc

[Ehsan Akhgari]

On Fri, Jun 23, 2017 at 4:19 AM, Chris Peterson <cpet...@mozilla.com>
wrote:

>
>
> On 6/23/17 12:17 AM, Ehsan Akhgari wrote:
>
> But to speak of a more direct measurement of performance, let's look at
> our progress on Speedometer V2
> <https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>.
> Today, I measured our progress so far on this benchmark by comparing
> Firefox 53, 54, 55.0b3 (latest beta as of this writing) and the latest
> Nightly, all x64 builds, on the reference hardware
> <https://www.amazon.com/dp/B01K1IO3QW>. This is the result (numbers are
> the reported benchmark score, higher is better):
>

> [image: Speedometer improvements]

>
>
> How do these Speedometer V2 scores map to the results on AWFY? AWFY shows
> many Speedometer sub-tests, but no score in the range of 70.21. AWFY
> machine #36 is the reference hardware.
>
> https://arewefastyet.com/#machine=36&view=breakdown&suite=speedometer-misc
>

Armen has been investigating the difference. The Speedometer benchmark is
extremely sensitive to anything else that is going on on the machine at the
time you are running the tests, see for example
https://bugzilla.mozilla.org/show_bug.cgi?id=1373396#c3 where he discovered
that turning off the "Shut off display after 15 minutes" setting improves
our benchmark score by about 10 points or so! I think there are other
investigations ongoing to dig into the remaining difference as well.

(Also note that there is a speed difference on Speedometer between Nightly
and Beta, where Nightly with the same code will be a bit slower than Beta
since some Nightly specific features do show up in Speedometer profiles
currently, for example things like bug 1375568 are currently Nightly only,
and there are also debugging checks like <
https://searchfox.org/mozilla-central/rev/3291398f10dcbe192fb52e74974b172616c018aa/ipc/chromium/src/base/pickle.h#26>
that show up a bit in profiles as well. I think that explains why 55.0b3
is scoring so high comparing to 56.0a1 there.)

Cheers,
--
Ehsan

[Armen Zambrano Gasparnian]

On Friday, 23 June 2017 10:05:42 UTC-4, Ehsan Akhgari wrote:
> On Fri, Jun 23, 2017 at 4:19 AM, Chris Peterson <cpet...@mozilla.com>
> wrote:
>
> >
> >
> > On 6/23/17 12:17 AM, Ehsan Akhgari wrote:
> >
> > But to speak of a more direct measurement of performance, let's look at
> > our progress on Speedometer V2
> > <https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>.

Ehsan, were you comparing against http://speedometer2.benj.me?
Or this http://browserbench.org/Speedometer/?
Or using the AWFY code? (It uses local proxy)

> > Today, I measured our progress so far on this benchmark by comparing
> > Firefox 53, 54, 55.0b3 (latest beta as of this writing) and the latest
> > Nightly, all x64 builds, on the reference hardware
> > <https://www.amazon.com/dp/B01K1IO3QW>. This is the result (numbers are
> > the reported benchmark score, higher is better):
> >
> > [image: Speedometer improvements]
> >
> >
> > How do these Speedometer V2 scores map to the results on AWFY? AWFY shows
> > many Speedometer sub-tests, but no score in the range of 70.21. AWFY
> > machine #36 is the reference hardware.
> >
> > https://arewefastyet.com/#machine=36&view=breakdown&suite=speedometer-misc
> >
>
> Armen has been investigating the difference. The Speedometer benchmark is
> extremely sensitive to anything else that is going on on the machine at the
> time you are running the tests, see for example
> https://bugzilla.mozilla.org/show_bug.cgi?id=1373396#c3 where he discovered
> that turning off the "Shut off display after 15 minutes" setting improves
> our benchmark score by about 10 points or so! I think there are other
> investigations ongoing to dig into the remaining difference as well.
>

Currently, machine #36 is testing against non-PGO builds from inbound (to catch regressions).
I'm looking into adding mozilla-central PGO builds to the mix.
I assume PGO builds can run slightly faster than non-PGO builds.

Once we have PGO builds we will be able to see if there are anymore machine configration changes required (I doubt it).

Direct link to the speedometer score on machine #36:
https://arewefastyet.com/#machine=36&view=single&suite=speedometer-misc&subtest=score

[Ehsan Akhgari]

On 06/26/2017 09:52 AM, Armen Zambrano Gasparnian wrote:
> On Friday, 23 June 2017 10:05:42 UTC-4, Ehsan Akhgari wrote:
>> On Fri, Jun 23, 2017 at 4:19 AM, Chris Peterson <cpet...@mozilla.com>
>> wrote:
>>
>>>
>>> On 6/23/17 12:17 AM, Ehsan Akhgari wrote:
>>>
>>> But to speak of a more direct measurement of performance, let's look at
>>> our progress on Speedometer V2
>>> <https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>.
> Ehsan, were you comparing against http://speedometer2.benj.me?

I used this instance for these measurements.
> Or this http://browserbench.org/Speedometer/?
This is still Speedometer V1, FWIW.

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Benjamin Bouvier]

2017-06-26 19:40 GMT+02:00 Ehsan Akhgari <ehsan....@gmail.com>:
> On 06/26/2017 09:52 AM, Armen Zambrano Gasparnian wrote:
>>

>> On Friday, 23 June 2017 10:05:42 UTC-4, Ehsan Akhgari wrote:
>>>
>>> On Fri, Jun 23, 2017 at 4:19 AM, Chris Peterson <cpet...@mozilla.com>
>>> wrote:
>>>
>>>>
>>>> On 6/23/17 12:17 AM, Ehsan Akhgari wrote:
>>>>
>>>> But to speak of a more direct measurement of performance, let's look at
>>>> our progress on Speedometer V2
>>>>
>>>> <https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>.
>>
>> Ehsan, were you comparing against http://speedometer2.benj.me?
>

> I used this instance for these measurements.
>>
>> Or this http://browserbench.org/Speedometer/?
>
> This is still Speedometer V1, FWIW.

Both websites indicate Speedometer V1 but are currently running Speedometer V2:
- The hosted version (under benj.me) is a direct clone of the Webkit
repository PerformanceTests/Speedometer with changes for running under
AWFY. I've manually updated the title's version number.
- For browserbench.org, see
https://github.com/WebKit/webkit/commit/ae7c2b286458944f61b33eea85fb89f263245b52#diff-0858c771e4c688cca20fc04d800816aa
. I've opened a pull request for updating their version number as
well.