About web-platform-tests in Intent to Ship

[Philip Jägenstedt]

Hi Blinketeers,

The launch process documentation mentions interop testing, and the “Intent to Ship” template says "Link to any web-platform-tests that cover this change, or explain why there aren’t any." in the Interoperability and Compatibility Risk section.

This is still fairly new and we haven't actually consistently been looking for it or asking about it. Starting roughly now, we'll try to do just that. This doesn't mean that web-platform-tests will be required, but if there aren't any we'd like to understand the reasons. For example, the API could be impossible to automate with current wpt infrastructure, or the implementation may have been going on for months before using web-platform-tests was practical.

The entry point documentation for web-platform-tests in Blink is:

https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md

On a related note, it looks like ~8% of CLs changing LayoutTests in Q1 were using wpt, let's see if that number grows over time :)

Happy testing!

[Mounir Lamouri]

On Wed, 22 Mar 2017, at 06:41, Philip Jägenstedt wrote:
> Hi Blinketeers,
>

> The launch process <http://www.chromium.org/blink/launching-features>

> documentation
> mentions interop testing, and the “Intent to Ship” template

> <https://docs.google.com/document/d/1vlTlsQKThwaX0-lj_iZbVTzyqY7LioqERU8DK3u3XjI/edit#bookmark=id.w8j30a6lypz0>

> says
> "Link to any web-platform-tests

> <https://github.com/w3c/web-platform-tests> that

> cover this change, or explain why there aren’t any." in the
> Interoperability and Compatibility Risk section.

To clarify, what are we requiring? Existence of WPT for the feature?
meaningful coverage of the feature with WPT? 100% coverage? or maybe
just all LayoutTests should be WPT and non-WPT LayoutTests should be
tagged with a reason? Those would be fairly different requirements when
sending an I2S but I guess when reviewing them, the work wouldn't be the
same either.

> This is still fairly new and we haven't actually consistently been
> looking
> for it or asking about it. Starting roughly now, we'll try to do just
> that.
> This doesn't mean that web-platform-tests will be required, but if there
> aren't any we'd like to understand the reasons. For example, the API
> could
> be impossible to automate with current wpt infrastructure, or the
> implementation may have been going on for months before using
> web-platform-tests was practical.

I wonder if idlharness tests shouldn't be required if the feature has
IDL changes. In most cases, at least part of the feature can be tested
with idlharness.

-- Mounir

> The entry point documentation for web-platform-tests in Blink is:
> https://chromium.googlesource.com/chromium/src/+/master/docs/testing/web_platform_tests.md
>
> On a related note, it looks like ~8% of CLs changing LayoutTests in Q1
> were
> using wpt

> <https://groups.google.com/a/chromium.org/d/msg/platform-predictability/uQzLZ3Fzkrc/ZMudLsYYAAAJ>,

> let's see if that number grows over time :)
>
> Happy testing!
>

> --
> You received this message because you are subscribed to the Google Groups
> "blink-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

On Mon, Mar 27, 2017 at 3:06 AM Mounir Lamouri <mou...@lamouri.fr> wrote:

On Wed, 22 Mar 2017, at 06:41, Philip Jägenstedt wrote:
> Hi Blinketeers,
>
> The launch process <http://www.chromium.org/blink/launching-features>
> documentation
> mentions interop testing, and the “Intent to Ship” template
> <https://docs.google.com/document/d/1vlTlsQKThwaX0-lj_iZbVTzyqY7LioqERU8DK3u3XjI/edit#bookmark=id.w8j30a6lypz0>
> says
> "Link to any web-platform-tests
> <https://github.com/w3c/web-platform-tests> that
> cover this change, or explain why there aren’t any." in the
> Interoperability and Compatibility Risk section.

To clarify, what are we requiring? Existence of WPT for the feature?
meaningful coverage of the feature with WPT? 100% coverage? or maybe
just all LayoutTests should be WPT and non-WPT LayoutTests should be
tagged with a reason? Those would be fairly different requirements when
sending an I2S but I guess when reviewing them, the work wouldn't be the
same either.

A hard requirement will be too much as long LayoutTests is more powerful than wpt thanks to internal testing APIs, but I would like to get to the point where an automated, shared test suite is the norm for all web platform features, whether existing or new.

Early on, many features will still need internal testing APIs, and there are probably undiscovered hurdles in using wpt that we need to adapt to. But perhaps this would do both now and in the future:

Of all the tests for web-exposed behavior, are any not in web-platform-tests? Please explain and link to bugs.

 The bugs could be:

• A web-platform-tests issue with the "infra" label explaining why a certain thing cannot be tested. (example)
• A spec issue for some change that would make it possible to test. (example)
• A Chromium issue to upstream some existing tests. (example)

Some judgement would still be required, but does that seem reasonable?

(I expect that we'd see certain kinds of problems blocking testing over and over again, which should be a signal to prioritize those problems.)

> This is still fairly new and we haven't actually consistently been
> looking
> for it or asking about it. Starting roughly now, we'll try to do just
> that.
> This doesn't mean that web-platform-tests will be required, but if there
> aren't any we'd like to understand the reasons. For example, the API
> could
> be impossible to automate with current wpt infrastructure, or the
> implementation may have been going on for months before using
> web-platform-tests was practical.

I wonder if idlharness tests shouldn't be required if the feature has
IDL changes. In most cases, at least part of the feature can be tested
with idlharness.

Yes, having an idlharness test would be nice. Certainly if it was a change to an existing method and there was an existing idlharness test, then updating that rather than making it fail would be a minimum.