PSA: Commit your WPT changes directly to LayoutTests/external/wpt

[Philip Jägenstedt]

Hi everyone,

At BlinkOn I presented on web-platform-tests

in Blink and beyond (video coming, I hope) and I thought I should also write a PSA:

Thanks to the hard work of jeffcarp@ and qyearsley@, we now have working 2-way sync between web-platform-tests and Blink.

The goal is that you can commit and forget, but see //docs/testing/web_platform_tests.md for a caveat around MANIFEST.json that currently applies.

Today we had 3 exports from different people, quite exciting! If you run into anything that makes using web-platform-tests less fun that regular LayoutTests, please let us know here or at platform-pr...@chromium.org.

Happy testing!

[Rick Byers]

Thanks Philip, Quinten, Jeff for making this happen, and to everyone else who helped and provided feedback on the design!

I just want to re-emphasize that making interop testing a first-class piece of blink engineering is strategically critical to chromium's mission to improve the web platform.  It would be crazy to expect the web to ever behave like a single platform without the engineering discipline of having a test suite shared by different implementations.  All the major engines are investing heavily in interop testing, and combined with the standards community shifting to being more test-focused, there's a ton of momentum right now to leverage into having a big impact on the fundamental nature of the web.

There's bound to be some bumps and frustration along the way (eg. needing to think about how your test behaves on non-Chrome browsers), but we expect that those costs will be more than offset by the benefit of sometimes having some of your tests already written for you and by a reduction in time spent on interop bugs post-ship.  As a result, I expect that sometime in 2017 we will add "has a good automated web-platform-test suite" to the launch requirements at "intent to ship" time.  Ideally most usage of LayoutTests will be replaced by WPT (and blink-specific unit testing frameworks like SimTests). We're not yet ready for that (eg. lack of standard automation APIs), but you should expect API owners to ask more often about the status of interop testing (as already mentioned in the launch guidelines).

Please do let us know if you have any concerns or feedback on how we can make it easier to achieve interoperability in the features you own!

Thanks,

   Rick

Today we had 3 exports from different people, quite exciting! If you run into anything that makes using web-platform-tests less fun that regular LayoutTests, please let us know here or at platform-predictability@chromium.org.

Happy testing!

[Simon Pieters]

On Tue, 07 Feb 2017 21:08:15 +0100, Rick Byers <rby...@chromium.org> wrote:

> Thanks Philip, Quinten, Jeff for making this happen, and to everyone else
> who helped and provided feedback on the design!
>

> I just want to re-emphasize that *making interop testing a first-class

> piece of blink engineering is strategically critical to chromium's
> mission

> to improve the web platform*. It would be crazy to expect the web to

> ever
> behave like a single platform without the engineering discipline of
> having
> a test suite shared by different implementations. All the major engines
> are investing heavily

> <https://docs.google.com/document/d/1lqOBTvaoTUIFuxK0IQL1UfXtybq7Kw8JxBvD6FZSdYE/edit#heading=h.aha7njnu40vz>

> in interop testing, and combined with the standards community shifting to
> being more test-focused

> <https://blog.whatwg.org/improving-interoperability>,

> there's a ton of momentum right now to leverage into having a big impact
> on
> the fundamental nature of the web.
>
> There's bound to be some bumps and frustration along the way (eg. needing
> to think about how your test behaves on non-Chrome browsers), but we
> expect
> that those costs will be more than offset by the benefit of sometimes
> having some of your tests already written for you and by a reduction in
> time spent on interop bugs post-ship. As a result, I expect that
> sometime
> in 2017 we will add "has a good automated web-platform-test suite" to the
> launch requirements at "intent to ship" time. Ideally most usage of
> LayoutTests will be replaced by WPT (and blink-specific unit testing
> frameworks like SimTests). We're not yet ready for that (eg. lack of
> standard automation APIs), but you should expect API owners to ask more
> often about the status of interop testing (as already mentioned in the
> launch

> guidelines <https://www.chromium.org/blink/launching-features>).

>
> Please do let us know if you have any concerns or feedback on how we can
> make it easier to achieve interoperability in the features you own!

This caused some confusion/unhappyness in this instance:

https://github.com/whatwg/html/issues/2249#issuecomment-278847830

This received 3 LGTMs in the intent thread. I will leave it to you to
discuss if something needs to change in the intent process or so to avoid
this kind of situation in the future.

> Thanks,
> Rick
>
> On Mon, Feb 6, 2017 at 10:06 PM, Philip Jägenstedt <foo...@chromium.org>
> wrote:
>
>> Hi everyone,
>>
>> At BlinkOn I presented on web-platform-tests

>> <https://docs.google.com/presentation/d/1s2Dick89wvJsuNJb4ia3pPt84NtMv8rZr0E_GFXJLrk/edit?usp=sharing>
>> in Blink and beyond
>> <https://docs.google.com/presentation/d/1s2Dick89wvJsuNJb4ia3pPt84NtMv8rZr0E_GFXJLrk/edit?usp=sharing>

>> (video
>> coming, I hope) and I thought I should also write a PSA:
>>
>> Thanks to the hard work of jeffcarp@ and qyearsley@, we now have working
>> 2-way sync between web-platform-tests and Blink.
>>
>> The goal is that you can commit and forget, but see
>> //docs/testing/web_platform_tests.md

>> <https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md>

>> for a caveat around MANIFEST.json that currently applies.
>>
>> Today we had 3 exports

>> <https://github.com/w3c/web-platform-tests/pulls?q=is%3Apr+is%3Aclosed+label%3Achromium-export>

>> from different people, quite exciting! If you run into anything that
>> makes
>> using web-platform-tests less fun that regular LayoutTests, please let
>> us

>> know here or at platform-pr...@chromium.org.
>>
>> Happy testing!
>>
>


--
Simon Pieters
Opera Software

[Rick Byers]

Thanks for pointing this out.  Sorry about the confusion.  There's always going to be some cases where we decide to land the blink changes for something before the spec has been updated, so we'll need some process for handling WPT in that case.  We'll work on figuring this out and update the docs with guidance.

Thanks,
   Rick

On Mon, Feb 6, 2017 at 10:06 PM, Philip Jägenstedt <foo...@chromium.org>
wrote:

Hi everyone,

At BlinkOn I presented on web-platform-tests
<https://docs.google.com/presentation/d/1s2Dick89wvJsuNJb4ia3pPt84NtMv8rZr0E_GFXJLrk/edit?usp=sharing>
in Blink and beyond
<https://docs.google.com/presentation/d/1s2Dick89wvJsuNJb4ia3pPt84NtMv8rZr0E_GFXJLrk/edit?usp=sharing> (video
coming, I hope) and I thought I should also write a PSA:

Thanks to the hard work of jeffcarp@ and qyearsley@, we now have working
2-way sync between web-platform-tests and Blink.

The goal is that you can commit and forget, but see
//docs/testing/web_platform_tests.md
<https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md>
for a caveat around MANIFEST.json that currently applies.

Today we had 3 exports
<https://github.com/w3c/web-platform-tests/pulls?q=is%3Apr+is%3Aclosed+label%3Achromium-export>

from different people, quite exciting! If you run into anything that makes
using web-platform-tests less fun that regular LayoutTests, please let us

know here or at platform-predictability@chromium.org.

Happy testing!

[Boris Zbarsky]

On 2/10/17 4:11 AM, Simon Pieters wrote:
> This caused some confusion/unhappyness in this instance:

I should note that these things happen. On the Mozilla side we try to
not land WPT changes that don't align with the current spec, but I'm
sure we've screwed that up a few times too.

One other mechanism we have in place for this sort of thing is
https://hg.mozilla.org/mozilla-central/file/tip/testing/web-platform/mozilla/README
-- a set of testharness tests that we haven't upstreamed yet for various
reasons. The failure mode here, of course, can be forgetting to upstream...

No easy solutions here that I see, unfortunately. Just needs vigilance
all around.

-Boris