Improved wpt-sync now running experimentally

[James Graham]

The new sync for web-platform-tests is now running experimentally. This
provides two way sync between the w3c/web-platform-tests repository on
GitHub and mozilla-central, so allowing gecko developers to contribute
to web-platform-tests using their normal gecko workflow, and ensuring
that we get all the upstream changes submitted by the community
including engineers at Google, Apple, and Microsoft.

The new code is intended to provide the following improvements over the
old periodic batch sync approach:

* Faster sync. The code to actually land changes to mozilla-central is
still undergoing testing, but the intent is that we can get at least one
wpt update per day once the system is fully operational.

* One bug per PR we downstream, filed in a component determined by the
files changed in the PR.

* One PR per bug we upstream. Currently this will be created when a
patch lands on inbound or autoland and should be merged when the patch
reaches central. In some hypothetical future world in which there's a
single entry point for submitting code to land in gecko (e.g.
phabricator) this will change so that the PR is created when the code is
submitted for review, so that upstream test results are available before
landing (see next point).

* Upstream CI jobs run on PRs originating from gecko repositories.
Previously we skipped upstream travis jobs on pushes we landed,
occasionally causing breakage as a result. Now these jobs are run on all
our pushes and the original bug should get a notification if the jobs fail.

* Notifications of notable changes introduced by upstream PRs. In
particular we will add a comment when tests that used to pass start to
not pass, when there are crashes or disabled tests, and for new tests
that fail. This notification happens in the bug for the sync, but there
is already an issue open to move things that obviously require attention
(e.g. crashes) into their own bug.

If you notice problems with the sync, please file an issue [1] or
complain in #wpt-sync on irc. The project team consists of:

* jgraham and maja_zf (development, primary contacts)
* AutomatedTester (project management)

Issues are not unanticipated at this time, so thanks in advance for your
patience as we work out the kinks in the system.

[1] https://github.com/mozilla/wpt-sync/issues

[Boris Zbarsky]

On 2/9/18 1:26 PM, James Graham wrote:
> The new code is intended to provide the following improvements over the
> old periodic batch sync approach:

Thank you. This is awesome.

-Boris

[Josh Bowman-Matthews]

On 2/9/18 1:26 PM, James Graham wrote:

> * One bug per PR we downstream, filed in a component determined by the
> files changed in the PR.

What does this mean exactly? What is the desired outcome of these bugs?

Cheers,
Josh

[James Graham]

They're tracking the process and will be closed when the PR lands in
central. They are used for notifying gecko developers about the incoming
change, and in particular contain the information about tests that went
from passing to failing, and other problems during the import.

They are not essential to the sync so if they end up not working well at
keeping people informed we can revisit the approach.

[smaug]

On 02/09/2018 10:39 PM, James Graham wrote:
> On 09/02/2018 19:59, Josh Bowman-Matthews wrote:
>> On 2/9/18 1:26 PM, James Graham wrote:
>>> * One bug per PR we downstream, filed in a component determined by the files changed in the PR.
>>
>> What does this mean exactly? What is the desired outcome of these bugs?
>
> They're tracking the process and will be closed when the PR lands in central. They are used for notifying gecko developers about the incoming change,
> and in particular contain the information about tests that went from passing to failing, and other problems during the import.

I guess I don't understand the bugmail. Most of the time I don't see any information about something failing. Am I supposed to look at the commit?
Or are new failures in bugmail like
"
Ran 2 tests and 44 subtests
OK : 2
PASS : 34
FAIL : 10
"

Are those 10 failures new failures, or failures from the test total?


-Olli

[James Graham]

On 12/02/2018 20:08, smaug wrote:
> On 02/09/2018 10:39 PM, James Graham wrote:
>> On 09/02/2018 19:59, Josh Bowman-Matthews wrote:
>>> On 2/9/18 1:26 PM, James Graham wrote:
>>>> * One bug per PR we downstream, filed in a component determined by
>>>> the files changed in the PR.
>>>
>>> What does this mean exactly? What is the desired outcome of these bugs?
>>
>> They're tracking the process and will be closed when the PR lands in
>> central. They are used for notifying gecko developers about the
>> incoming change, and in particular contain the information about tests
>> that went from passing to failing, and other problems during the import.
>
> I guess I don't understand the bugmail. Most of the time I don't see any
> information about something failing. Am I supposed to look at the commit?
> Or are new failures in bugmail like
> "
> Ran 2 tests and 44 subtests
> OK     : 2
> PASS   : 34
> FAIL   : 10
> "
>
> Are those 10 failures new failures, or failures from the test total?

That's the total failures. If that's all you see then nothing fell into
one of the predefined categories of badness that get extra details added
to the comment. If there is some information that you think should be
present but is actually missing, please file an issue.