Merging Quantum Render into mozilla-central

[Kartikaya Gupta]

(bcc: dev-platform, dev-tech-gfx. Please keep discussion on dev-tree-management)

Work for the Quantum Render ("QR") project has been happening on the
graphics branch [1]. We've been doing regular merges from m-c to the
graphics branch, but the merge conflicts have slowly been getting more
complex as the two trees diverge. If we could merge graphics back to
m-c periodically it would help reduce this burden. I'd like to get
sheriffs to sign off on this, and give anybody else a chance to
comment as well.

What I would like to do is merge the changes from the graphics branch
back into m-c, but continue to keep the graphics branch as a "working
branch" for the QR project. We would continue doing daily merges from
m-c to the graphics branch, but only merge back from graphics to m-c
occasionally (I'm thinking around once a week or so). The graphics
branch would remain sheriffed by the graphics team.

As part of this, I would like to enable QR build jobs (which are
normal linux64/osx64/win64 builds with --enable-webrender in the
mozconfig) as tier-1, and linux64 QR test jobs as tier-2. I don't
expect that having the builds tier-1 would be a burden on platform
developers, as almost all of the new c++ code is compiled regardless
of the --enable-webrender flag, so developers will catch compile
errors with their usual workflow. Tests being tier-2 seems like an
acceptable compromise, as it doesn't block platform developers but
also gives us an early warning of changes that potentially affect QR.
In practice, most test failures we've experienced are simply new
reftests that need to be annotated with fails-if(webrender).

As QR builds would be tier-1, they would need to run on all
integration branches (including m-i and autoland), but I think QR test
jobs could be limited to running on m-c, to reduce the load on our
infra. Right now there's not a lot of QR test jobs anyway but we're
actively greening up more test suites and getting them running on the
QR builds.

If you're interested in more details on how we would go about doing
the merge, please see [2]. Any comments or suggestions are welcome.

Cheers,
kats

[1] https://hg.mozilla.org/projects/graphics
[2] https://groups.google.com/d/msg/mozilla.dev.tech.gfx/Foxlm-c-Md8/y6khtK1CCwAJ

[Lawrence Mandel]

Hi kats,

We need to be cautious about the extra cost being incurred by adding new
build and test jobs. We made a decision not to budget for Quantum specific
builds and tests. Relatively inexpensive additions should be fine. Without
knowing more, I'm concerned about the extra cost of executing new tier1
builds on all branches and executing new batches of tests if they're
large/expensive. I'm not trying to say that we can't do anything but I
would like to understand the cost of what's being proposed before we decide
on what we can do.

Lawrence

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management
>

[Kartikaya Gupta]

Hi Lawrence,

For the time being the set of extra jobs is pretty small. Right now on
the graphics branch the extra jobs that we are doing for QR are:
- linux64 QR builds with cppunit, gtest, jittest, reftest,
reftest-no-accel tests
- macosx64 QR builds
- win64 QR builds

Each of the above is run in debug and opt versions. At a minimum I'd
like to run all of this on m-c (so ~3-4 pushes a day or so). Running
the build jobs on the integration branches and making them tier-1
would be nice, but I'd be ok with just leaving them as tier-2 on m-c,
because generally I don't think they will break without breaking other
builds that are already covered on the integration branches. So I
guess it's a question of how much the above-listed set of jobs costs
and if it's ok to run it on m-c. Note that the set will probably
expand slowly as we bring more tests online.

However, longer-term I guess it would be good to figure out the
testing strategy for QR. I anticipate that we'll remove the
compile-time switch in the not-too-distant future, and leave a runtime
switch instead. That will eliminate the "build" jobs. However we'll
still want to keep test coverage of both QR and non-QR codepaths, much
like we have e10s and non-e10s test coverage. We can probably limit it
to test suites that exercise the graphics stack more heavily, but it
will probably still be a nontrivial amount of extra infra load. I
don't know what to do about that, but I'm open to suggestions.

Cheers,
kats

[Kim Moir]

I've opened https://bugzilla.mozilla.org/show_bug.cgi?id=1335197 to discuss
this issue of enabling graphics builds + tests on m-c.(Allow us to address
capacity issues)

Kim

[Gregory Szorc]

On Mon, Jan 30, 2017 at 11:56 AM, Kartikaya Gupta <kgu...@mozilla.com>
wrote:

> (Explicitly forwarding to dev-tech-gfx and dev-platform, since
> apparently bcc'ing lists gets messages stuck in moderation. Please
> reply on dev-tree-management. Sorry for my mail-fail!)

>
>
> ---------- Forwarded message ----------
> From: Kartikaya Gupta <kgu...@mozilla.com>
> Date: Mon, Jan 30, 2017 at 2:22 PM
> Subject: Merging Quantum Render into mozilla-central
> To: dev-tree-...@lists.mozilla.org
>
>
> (bcc: dev-platform, dev-tech-gfx. Please keep discussion on
> dev-tree-management)
>
> Work for the Quantum Render ("QR") project has been happening on the
> graphics branch [1]. We've been doing regular merges from m-c to the
> graphics branch, but the merge conflicts have slowly been getting more
> complex as the two trees diverge.

I'm glad you have rediscovered why project branches aren't great long-term
solutions and that development should happen as close to mozilla-central as
possible :)

> If we could merge graphics back to
> m-c periodically it would help reduce this burden. I'd like to get
> sheriffs to sign off on this, and give anybody else a chance to
> comment as well.
>
> What I would like to do is merge the changes from the graphics branch
> back into m-c, but continue to keep the graphics branch as a "working
> branch" for the QR project. We would continue doing daily merges from
> m-c to the graphics branch, but only merge back from graphics to m-c
> occasionally (I'm thinking around once a week or so). The graphics
> branch would remain sheriffed by the graphics team.
>
> As part of this, I would like to enable QR build jobs (which are
> normal linux64/osx64/win64 builds with --enable-webrender in the
> mozconfig) as tier-1, and linux64 QR test jobs as tier-2. I don't
> expect that having the builds tier-1 would be a burden on platform
> developers, as almost all of the new c++ code is compiled regardless
> of the --enable-webrender flag, so developers will catch compile
> errors with their usual workflow. Tests being tier-2 seems like an
> acceptable compromise, as it doesn't block platform developers but
> also gives us an early warning of changes that potentially affect QR.
> In practice, most test failures we've experienced are simply new
> reftests that need to be annotated with fails-if(webrender).
>

This is basically what I did for Stylo in bug 1318200. Extending that
solution to work for QR should be trivial. Although it appears from [2]
that most of that work is already done on the graphics repo, just without
build peer review. So, uh, you may want to verify your automation code
matches what is used for Stylo before submitting that review request
because if it doesn't, it will likely get r- and the review will be "make
it more like what Stylo does."

When you say "linux64 QR test jobs," what exactly does that mean? Is there
a new test harness for a new class of tests? Or, is this running a large
subset of existing tests just with QR enabled? If so, which ones? (If we're
running a variation of existing tests, we could run into capacity concerns.)

>
> As QR builds would be tier-1, they would need to run on all
> integration branches (including m-i and autoland), but I think QR test
> jobs could be limited to running on m-c, to reduce the load on our
> infra. Right now there's not a lot of QR test jobs anyway but we're
> actively greening up more test suites and getting them running on the
> QR builds.
>

As Lawrence said, capacity could be problematic. Generally speaking, it is
easier to scale Linux than Windows than OS X. And, builds are easier to
scale than tests because the number of machine hours spent running tests
dwarfs time running builds. What I'm trying to say is that this may
ultimately result with a tiered rollout where some platforms are higher
tiered than others for capacity reasons.

>
> If you're interested in more details on how we would go about doing
> the merge, please see [2]. Any comments or suggestions are welcome.
>
>

My understanding of the development model on the graphics repo is that many
of the commits were throwaway. If this is indeed the case and the history
doesn't provide much value, I'd seriously consider introducing a single,
non-merge changeset in mozilla-central for this "import" instead of
polluting history with value-less changesets.

Also, you could consider going the model of Stylo, which will have commits
cherry picked from upstream into mozilla-central without explicit merges.
That keeps history in mozilla-central much cleaner.

> Cheers,
> kats
>
> [1] https://hg.mozilla.org/projects/graphics
> [2] https://groups.google.com/d/msg/mozilla.dev.tech.gfx/
> Foxlm-c-Md8/y6khtK1CCwAJ
> _______________________________________________

> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Chris Peterson]

On 1/30/2017 3:23 PM, Gregory Szorc wrote:
> As Lawrence said, capacity could be problematic. Generally speaking, it is
> easier to scale Linux than Windows than OS X. And, builds are easier to
> scale than tests because the number of machine hours spent running tests
> dwarfs time running builds. What I'm trying to say is that this may
> ultimately result with a tiered rollout where some platforms are higher
> tiered than others for capacity reasons.

For comparison, the Stylo team has a project branch, but is adding just
one Stylo build+test job (linux64-stylo) to run on merges to
mozilla-central (bug 1330414).

> Also, you could consider going the model of Stylo, which will have commits
> cherry picked from upstream into mozilla-central without explicit merges.
> That keeps history in mozilla-central much cleaner.

Yes. The Stylo team has a project branch with some temporary test code
commits, but Stylo feature work only lands upstream in mozilla-central
or servo's GitHub repo.

[Kartikaya Gupta]

On Mon, Jan 30, 2017 at 6:23 PM, Gregory Szorc <g...@mozilla.com> wrote:
> I'm glad you have rediscovered why project branches aren't great long-term
> solutions and that development should happen as close to mozilla-central as
> possible :)

It was never meant to be a long-term solution, but yes, the calculus
of costs vs benefits is shifting now and it makes sense to move it
closer to m-c.

>> As part of this, I would like to enable QR build jobs (which are
>> normal linux64/osx64/win64 builds with --enable-webrender in the
>> mozconfig) as tier-1, and linux64 QR test jobs as tier-2. I don't
>> expect that having the builds tier-1 would be a burden on platform
>> developers, as almost all of the new c++ code is compiled regardless
>> of the --enable-webrender flag, so developers will catch compile
>> errors with their usual workflow. Tests being tier-2 seems like an
>> acceptable compromise, as it doesn't block platform developers but
>> also gives us an early warning of changes that potentially affect QR.
>> In practice, most test failures we've experienced are simply new
>> reftests that need to be annotated with fails-if(webrender).
>
>
> This is basically what I did for Stylo in bug 1318200. Extending that
> solution to work for QR should be trivial. Although it appears from [2] that
> most of that work is already done on the graphics repo, just without build
> peer review.

Yeah, I made changes to the in-tree taskcluster configs in the
graphics branch that was roughly modeled after the stylo taskcluster
changes. I had to go a bit beyond stylo to get the win64 and macosx64
builds as well but the changes are roughly analogous.

> So, uh, you may want to verify your automation code matches
> what is used for Stylo before submitting that review request because if it
> doesn't, it will likely get r- and the review will be "make it more like
> what Stylo does."

I'm expecting that we might have to make minor tweaks to the
taskcluster configs based on the review comments but hopefully nothing
major. (And even if they are major, it's a one-time thing, I can deal
with it.)

> When you say "linux64 QR test jobs," what exactly does that mean? Is there a
> new test harness for a new class of tests? Or, is this running a large
> subset of existing tests just with QR enabled? If so, which ones? (If we're
> running a variation of existing tests, we could run into capacity concerns.)

I just mean the existing test jobs that we've enabled to run on the
linux64 QR builds. No new harness. The tests we have enabled so far
can be seen at https://hg.mozilla.org/projects/graphics/file/12c5788aa7a0/taskcluster/ci/test/test-sets.yml#l72
(or take a look at the graphics branch on TH).

>> As QR builds would be tier-1, they would need to run on all
>> integration branches (including m-i and autoland), but I think QR test
>> jobs could be limited to running on m-c, to reduce the load on our
>> infra. Right now there's not a lot of QR test jobs anyway but we're
>> actively greening up more test suites and getting them running on the
>> QR builds.
>
>

> As Lawrence said, capacity could be problematic. Generally speaking, it is
> easier to scale Linux than Windows than OS X.

So far we're only running on tests on Linux because TaskCluster isn't
really working for the other desktop platforms. (At least not that I'm
aware of).

> And, builds are easier to
> scale than tests because the number of machine hours spent running tests
> dwarfs time running builds. What I'm trying to say is that this may
> ultimately result with a tiered rollout where some platforms are higher
> tiered than others for capacity reasons.

That seems fair. We can prioritize test suites so that e.g. Linux runs
a larger set and Windows runs just the really graphics-critical ones.
But at the moment I have my hands full with just getting the Linux
suites turned on, so adding jobs on other platforms is probably at
least a month away.

>> If you're interested in more details on how we would go about doing
>> the merge, please see [2]. Any comments or suggestions are welcome.
>>
>
> My understanding of the development model on the graphics repo is that many
> of the commits were throwaway. If this is indeed the case and the history
> doesn't provide much value, I'd seriously consider introducing a single,
> non-merge changeset in mozilla-central for this "import" instead of
> polluting history with value-less changesets.

There aren't actually that many throwaway commits in the graphics
mercurial repo. Prior to using the graphics branch, we were doing
development on github, and that *did* have a lot of throwaway commits.
When I migrated the code from github to the graphics branch I squashed
it all into four commits (see bug 1317774). At this point we have ~320
non-merge commits on the graphics branch that would get merged over to
m-c, and I feel like the history there is mostly pretty useful to
have.

> Also, you could consider going the model of Stylo, which will have commits
> cherry picked from upstream into mozilla-central without explicit merges.
> That keeps history in mozilla-central much cleaner.

I think the boat on keeping m-c free of merge commits is probably out
of the solar system by now :) Seriously though, I'm not sure what
value doing cherry-picks instead of merges would provide. It kind of
makes sense for Stylo because of the integration with upstream servo
and how commits have to be gated against both m-c and servo, but we
don't have that restriction for QR. I think merges are less work than
doing cherry-picks overall. I do plan on doing a few initial
cherry-picks of stuff like the taskcluster changes and build system
stuff, but that's more because it needs proper review from
non-graphics people. Once the initial merge is done I don't see any
value provided by cherry-picking.

Cheers,
kats

[Gregory Szorc]

This set doesn't seem too bad. reftests look to have the largest cost. The
capacity impact from those tests will be noticeable, but it's a lot better
than say if mochitests were also being run.

Out of curiosity, is there a way to limit tests within suites to just those
impacted by QR? For example, there are tons of xpcshell tests not at all
related to anything Quantum. Test manifests support "tags" and it is
possible to configure automation to only run tests having a certain flag.
In a TaskCluster world, I would entertain the idea of splitting various
suites into separate jobs grouped by tags. This will soon be technically
achievable by integrating the TaskCluster "decision" task / task group with
build system metadata.

I didn't realize the GitHub history was rewritten. That alleviates most of
my concerns. (Although I saw a note about a 2nd copy of freetype. Stuff
like that is annoying, but possibly tolerable.)

>
> > Also, you could consider going the model of Stylo, which will have
> commits
> > cherry picked from upstream into mozilla-central without explicit merges.
> > That keeps history in mozilla-central much cleaner.
>
> I think the boat on keeping m-c free of merge commits is probably out
> of the solar system by now :) Seriously though, I'm not sure what
> value doing cherry-picks instead of merges would provide. It kind of
> makes sense for Stylo because of the integration with upstream servo
> and how commits have to be gated against both m-c and servo, but we
> don't have that restriction for QR. I think merges are less work than
> doing cherry-picks overall. I do plan on doing a few initial
> cherry-picks of stuff like the taskcluster changes and build system
> stuff, but that's more because it needs proper review from
> non-graphics people. Once the initial merge is done I don't see any
> value provided by cherry-picking.
>

The boat is sailing towards 100% of landings going through Autoland (the
service and the repo). At that point, the autoland repo is essentially Try
and there are no integration repos, so no merges. But for the short term,
merges from the graphics repo are acceptable if Stylo's workflow doesn't
make sense. At the end of the day, I care more about enabling developers to
be productive. And you are correct that merges are easier than
cherry-picks. So I'm fine with merges for the foreseeable future (you'll
still need to run that by Sheriffs if you are expecting them to do any
merging).

[Mike Hommey]

On Mon, Jan 30, 2017 at 04:35:19PM -0800, Gregory Szorc wrote:
> (Although I saw a note about a 2nd copy of freetype. Stuff
> like that is annoying, but possibly tolerable.)

Huh, please no.

Mike

[Kartikaya Gupta]

On Mon, Jan 30, 2017 at 7:35 PM, Gregory Szorc <g...@mozilla.com> wrote:
> This set doesn't seem too bad. reftests look to have the largest cost. The
> capacity impact from those tests will be noticeable, but it's a lot better
> than say if mochitests were also being run.

I do plan on getting mochitests running on the graphics branch, but we
probably don't need them running on m-c. Maybe just the webgl ones.

> Out of curiosity, is there a way to limit tests within suites to just those
> impacted by QR? For example, there are tons of xpcshell tests not at all
> related to anything Quantum. Test manifests support "tags" and it is
> possible to configure automation to only run tests having a certain flag. In
> a TaskCluster world, I would entertain the idea of splitting various suites
> into separate jobs grouped by tags. This will soon be technically achievable
> by integrating the TaskCluster "decision" task / task group with build
> system metadata.

This is an interesting idea. However it's not obvious to me how to
distinguish which tests (within a suite) are impacted by QR and which
are not. Certainly new tests that we add specifically for testing QR
codepaths can be turned off for non-QR builds. However short of going
through each test, I don't know how to determine on a per-test basis
which tests exercise QR codepaths and which ones don't. It's probably
better to just leave certain suites off entirely if they're not
particularly graphics-related.

> I didn't realize the GitHub history was rewritten. That alleviates most of

> my concerns. (Although I saw a note about a 2nd copy of freetype. Stuff like

> that is annoying, but possibly tolerable.)

So there was a 2nd copy of freetype in the tree for a short period of
time, and has since been removed (see bug 1318745). It is in the
graphics branch history, and removing it is the biggest argument in
favor of tampering with the graphics branch history before the merge.
However I did check the size of the .hg folder of m-c and compared it
to m-c + graphics, and it seemed to be about the same, so at least
from a mercurial-repo-size consideration it shouldn't be a problem.
(The second copy of freetype was mostly the same as the existing copy
of freetype, so I believe mercurial deduplicated it away).

>
> The boat is sailing towards 100% of landings going through Autoland (the
> service and the repo). At that point, the autoland repo is essentially Try
> and there are no integration repos, so no merges. But for the short term,
> merges from the graphics repo are acceptable if Stylo's workflow doesn't
> make sense. At the end of the day, I care more about enabling developers to
> be productive. And you are correct that merges are easier than cherry-picks.
> So I'm fine with merges for the foreseeable future (you'll still need to run
> that by Sheriffs if you are expecting them to do any merging).

Sounds good, thanks.

[Mike Hommey]

On Tue, Jan 31, 2017 at 09:58:54AM -0500, Kartikaya Gupta wrote:
> > I didn't realize the GitHub history was rewritten. That alleviates most of
> > my concerns. (Although I saw a note about a 2nd copy of freetype. Stuff like
> > that is annoying, but possibly tolerable.)
>
> So there was a 2nd copy of freetype in the tree for a short period of
> time, and has since been removed (see bug 1318745). It is in the
> graphics branch history, and removing it is the biggest argument in
> favor of tampering with the graphics branch history before the merge.
> However I did check the size of the .hg folder of m-c and compared it
> to m-c + graphics, and it seemed to be about the same, so at least
> from a mercurial-repo-size consideration it shouldn't be a problem.
> (The second copy of freetype was mostly the same as the existing copy
> of freetype, so I believe mercurial deduplicated it away).

FWIW, mercurial doesn't deduplicate files. It's just that freetype is
rather small.

Mike