Testing the untestable

[Philip Jägenstedt]

Hi Blink test automateers,

With web-platform-tests playing an increasingly large role in our testing efforts, many people are trying to automate the testing of their features and looking for ways to make that part of web-platform-tests.

Here are the recent efforts that I'm aware of:

• Web Input Automation
  
• public-test-infra: UserAgent-specific files in Web Platform Tests  (continuation)
• platform-predictability: Chromium Architecture for exposing web-platform testing APIs
• web-platform-tests: Initial pass of WebVR 1.1 conformance tests (background)
• public-test-infra: Specifications with matching test APIs (proposal)
• public-webappsec: [permissions] Defining a testing API

There are probably more that haven't caught my attention.

The short story is that various people are trying to do one of two things:

• Define extensions to WebDriver, which is the cross-vendor protocol for launching and controlling a browser. This is the approach taken for input automation and probably permission prompts. (Such an WebDriver APIs could be wrapped with JavaScript so that a test could drive itself.)
• Define test-only APIs that are used in web-platform-tests and could be exposed by a command line flag, somewhat similar to LayoutTests' internals.* APIs, but cross-browser. This is what was proposed for Bluetooth and USB.

So far no tests have landed in web-platform-tests with either approach. The WebDriver approach would be uncontroversial and is blocked on people to do the per-feature work. For the test-only API approach, there are some concerns, but Microsoft proposed such a solution for WebVR, so I think it will eventually happen for some feature.

If you're trying to automate wpt for your feature, what are you to make of this situation? I would suggest:

• Find your peers at other browser vendors and discuss with them. Shared automated tests is the whole point, and will require other vendors to be on board.
• Explore WebDriver first when the size of the API would be moderate.
• If both we and at least one other vendor would prefer a test-only API, work out a shared proposal and send it to public-test-infra. (Also let me know, I'll be very excited.)

Hope that helps.

[Philip Jägenstedt]

I should also add that I'd have these goals for any solution:

• APIs, whether in WebDriver or JavaScript are as clearly defined as the feature under test itself.
• The API will eventually be available in the standard releases of all browsers, so that tests do not need special builds. This will be important for running continuous integration tests on wpt PRs and on a wpt dashboard, both of which are ongoing work.
• Consequently: The APIs don't add unreasonably to the binary size, since we want lots of them.

[Vincent Scheib]

Adding tests to WebDriver will cost binary size. I don't see a way around that. 

[Reilly Grant]

I think the discussion of WebDriver vs. test-only API is actually a bit of a distraction. The reason for this is that in Chrome both of these are implemented as layers on top of something else. WebDriver is built on top of the Chrome DevTools remote debugging protocol. The test-only APIs are built on top of Mojo interfaces exposed only when Chrome is run in a special testing mode.

Long term Web Bluetooth and WebUSB would like to build high-quality DevTools features to assist developers using our APIs. For example, a view similar to the Network tab showing a timeline of device requests and responses. DevTools Extensions could also be written to emulate particular devices for testing purposes. This interface could be used build a high-quality WebDriver interface for automating tests of code using the WebUSB API. I am reluctant to devote the resources necessary to implementing this solution while Web Bluetooth and WebUSB are unproven APIs as it is the substantially more complex option.

The test-only API on the other hand is a much more limited API surface. Right now it can be implemented in a browser-specific way which has much lower overhead. When these APIs are more proven and automation support has been added to WebDriver the test-only API can be polyfilled on top of the WebDriver API.

[Philip Jägenstedt]

Neither do I, really, the question is how much is "unreasonable" and if the binary size increase is very different depending on approach.

ChromeDriver is actually a separate download, maybe it's possible to shift most of the cost to that? +joh...@chromium.org?

[Philip Jägenstedt]

It sounds like you think it is quite consequential which of the approaches is taken here, and that for Bluetooth and USB a test-only API is the best place to start. That makes sense, and I didn't mean to imply that Bluetooth and USB are cases where "the size of the [WebDriver] API would be moderate."

I'm not really happy in general with waiting until another vendor approves of the test API before sharing tests, as we don't block on the specs themselves having multi-vendor buy-in before writing tests. If we always did, then chances are that when people asked us to upstream the tests, nobody would have the time for it.

I hope that once we've proven that test-only APIs work in some area with multi-vendor interest, then things will go more smoothly for new and experimental things.