Writing Web Platform Tests

[David Bokan]

Hi all,

With our increased focus on public web platform tests, I figure the viewport API should use those as well but I have a few questions:

Is this what we've been talking about: https://www.chromium.org/blink/importing-the-w3c-tests (and are these instructions up-to-date) or is it something else?

Is there some kind of driver to assist with testing, something akin to our EventSender? In particular, there's parts of the viewport API that can't be scripted (such as pan scrolling when zoomed in) so at least some of our tests require inserting user input. I've heard WebDriver thrown around a bunch, presumably it plays a part here?

If the link above is correct, it seems like we import these tests into our LayoutTests directory. Does this mean that all our testing for a particular feature could be written as these web platform tests or is these meant as an "in addition" test of interoperability? 

Are these tests appropriate for unshipped features? I suppose our layout tests will run with "experimental" features turned on. But if there's web platform tests that we don't currently support they can't cause our tests to fail, right? i.e. if Apple added a tests for animation timebase tomorrow, it wouldn't cause redness on our builders. Does that mean that we don't cause redness based on these tests? Or do we opt some tests in? 

Thanks,
David

[Tim Dresser]

https://www.chromium.org/blink/importing-the-w3c-tests is talking about the right thing. I can't speak to its accuracy.

There are a few different categories of layout tests.

We have js-tests, which are Chrome only.

We have testharness.js tests, which are Chrome only.

We have web-platform-tests, which use testharness.js, don't use any Chrome specific features, and have been upstreamed to the w3c repo.

Ideally all of our tests would be web-platform-tests. If we have adequate web-platform-test coverage, we don't need any other tests. In some cases, where we've got a test to make sure we don't have some implementation specific quirk, we might want a few testharness.js tests which aren't upstreamed, as they wouldn't make sense for other browsers.

Right now, we can't synthesize input in web-platform-tests. There's an open bug on this here: https://github.com/w3c/web-platform-tests/issues/2161.

My recommendation for the moment would be to write testharness.js tests, using the gpu benchmarking extension.

The gpu benchmarking extension isn't used in layout tests currently, but when we get webdriver support, input will be synthesized approximately the same way in webdriver as it is in the gpu benchmarking extension.

For gpu benchmarking, run with the --enable-gpu-benchmarking flag - we should probably always run layout tests with this flag for now.

Then chrome.gpuBenchmarking has a bunch of methods for simulating user input.

These methods are defined here.

Here's an example usage: telemetry's pinch gesture.

[Tim Dresser]

Layout tests need to use the browser specific internals api to turn on experimental features, so I don't think we could upstream tests of experimental features.

Currently the import process is pretty manual, and I think we just clone some of the web-platform-tests.

Tim

[David Bokan]

Got it, thanks for the info.

Ok, I'll add tests using testharness.js with an eye to eventually upstreaming them so that once it leaves the experimental state we should be easily able to push them upstream.

Thanks!

[Rick Byers]

+tkent who is working on improving the situation with web-platform-test (the ultimate goal is that you can just checkin your interop tests along with your feature and it'll get magically pushed to the w3c repo, presumably with some checkin failure if you're depending on some blink-specific testing APIs).

In addition to Tim's advice for input-tests, I'd add that when your test doesn't require any synthetic input you should totally try to get it into web-platform-tests and import to blink from there.  That's more manual than we'd like at the moment (and WPT reviews can sometimes be slow/painful so you don't want to block on that per se), but still totally valuable.  And to answer your other question: no we don't want to duplicate testing between WPT and LayoutTests (which is a main reason we haven't imported more WPT tests wholesale).  Please write testharness tests (with the ultimate goal of WPT upstreaming) for substantial web-exposed behavior (stuff the spec takes a position on and isn't just an implementation detail), but feel free to also write traditional layout tests depending on eventSender, window.internals, gpuBenchmarking extention etc. for stuff that's really more of an implementation/unit test.  I expect we'll get this codified and polished into an easy / consistent process across the team sometime in 2016.

Thanks for asking about this, interop testing is really important!

Rick