WebRender status update

[Kartikaya Gupta]

Hi all,

I just wanted to make sure everybody working on WR is in sync with
respect to what the next major milestones are and what work to
prioritize.

===========
First milestone - WebRender Shield Study
===========

I had a meeting yesterday with some people where we hammered out the
plan for the Shield study. This is where we enable WebRender for 2
weeks on a sample of the Nightly population. The sample will be taken
from our V1 target population (Windows 10 + NVidia graphics) and be
large enough that we have confidence in the results we get. In order
to do the Shield study, we need to:

(1) Ensure that the relevant telemetry probes are in place for
comparing relevant metrics between the WR and non-WR codepaths. I just
landed one probe in bug 1463911 and Ryan will be adding another in bug
1470528.
(2) Update the Product Hypothesis Doc with the probes we want to
measure and some other details. I'll be doing that this week and
coordinating with Thomas Elin to get the necessary review on that.
(3) Get QA to sign off on the experiment changes (i.e. they run with
WR enabled and make sure it's not gonna blow up). I'll coordinate with
Thomas Elin to get this done.

The plan right now is to get these things done before an experiment
review meeting on July 11th; assuming all goes well the 2-week
experiment will launch on July 16th. Other than Ryan getting the
telemetry probe in I don't think anybody else has anything to do for
this milestone.

===========
Second milestone - WebRender on Nightly
===========

This is where we turn the "gfx.webrender.all.qualified" pref on by
default on the Nightly channel. This will enable WR by default for our
V1 target population. In order for this to happen, we need to do two
things:

(1) Deal with all the P1 blockers of bug 1386665 (stage-wr-nightly).
There are currently 16 open bugs on this list. If you're looking for
stuff to work on, stuff from this list would be the top priority. If
you think there's a bug on this list that shouldn't be there, or if
there's a bug that's not on the list but should be, please make the
changes (leave a comment in the bug for justification). Please assign
to yourself a bug if you're working on it, and take bugs (with
permission) that are assigned but aren't being actively worked on.
(2) Deal with any issues that come up as a result of the Shield study.
If the crash rate is too high, for example, or there's severe
regressions as measured by telemetry, those will need to be addressed
before we can turn it on in Nightly. We won't know that until after
the study is done though.

The goal is to get to this milestone while 63 is on Nightly. We just
started Nightly 63 yesterday, and the next merge is scheduled for
September 4th, so we have ~10 weeks all told. If we prioritize that P1
list, hopefully we can get them mostly under control in the next few
weeks and leave some time to deal with the issues coming out of the
study.

===========
Third milestone - WebRender riding the trains
===========

This is where we let gfx.webrender.all.qualified=true ride the trains.
It's not yet clear to me if we'll let it ride to Beta and Release on
the same train, or let it ride to Beta and hold it there for another
cycle or two before it goes out to Release. But there's a lot of work
that needs to happen before it even goes to Beta. We have a release
criteria document at [1] which lists all of the criteria for both Beta
and Release. I'm not going to repeat those here, but feel free to look
at the doc. The plan is that anything that violates the release
criteria is going to be tracked using a P1 bug blocking bug 1386669
(stage-wr-trains). This includes things like Talos regressions,
unfixed tests, telemetry probe regressions, crashers, and general
bugs. Some of these things (like telemetry probe regressions) we don't
have bugs for yet because we don't have the data. Others (like unfixed
tests) we have some bugs for, but not complete. I'll try to ensure
that we have bugs on file for everything. There's a lot of work that
needs to be done to hit this milestone still.

[1] https://docs.google.com/document/d/1zs5b-hAXnIxvl_acGUjibSSeT4ftI_3TIqAoePgCtBI

Cheers,
kats