Chromium Architecture for exposing web-platform testing APIs

[Vincent Scheib]

I'd like to drive towards a chromium wide recommendation for new web platform tests being authored in the near term that will need to add non-trivial testing API. Yes or no to pursuing Web Driver now:

On public-test-infra 

 "UserAgent-specific files in Web Platform Tests"

 https://lists.w3.org/Archives/Public/public-test-infra/2017JanMar/thread.html#msg30

 https://lists.w3.org/Archives/Public/public-test-infra/2017AprJun/thread.html#msg0

ortuno@ proposed a method to implement cross-browser testing APIs by user-agents injecting agent specific testing API implementations. Web platform tests would use specified testing APIs, but the implementation would be user-agent specific.

On platform-predictability "Mocking API for Bluetooth Devices" we also came to the conclusion that most options of adding testing APIs would increase the binary size of the browser by at least 100/200kb -- and therefore we will not ship testing APIs to stable browser versions. In the long term, we may add that overhead implicitly in the servicification process and could revisit by using that API over mojo. 

 

Web Driver has been raised as a specific testing API surface to use. It has merit as that API is mature and broadly used. 

My concern is that authoring tests directly to Web Driver protocol will require exposing Web Driver testing APIs, something we don't want to ship to all users (where I believe all our web driver implementation right now is shipped to stable channel), and reduces user-agent flexibility.

ortuno@'s proposal of a web IDL for the testing interface allows user-agents to change their implementation of the test API over time. It may be very shallow - e.g. in chrome living in the renderer only. Or, very deep, e.g. controlling actual device hardware and testing all the way from web through the browser, drivers, and OS. It also permits the use of Web Driver for user agents interested in that route.

[Reilly Grant]

Another concrete example, based on ortuno@'s proposal, is a patch I have currently out for review that uses a fictitious "WebUSB Test" API to upstream WebUSB tests to WPT. It includes a polyfill that is injected when these tests are run in Chromium. If this patch lands I will write a spec for this API so that other vendors can implement (or polyfill) it.

https://codereview.chromium.org/2789723003/

[Philip Jägenstedt]

Thanks for pushing on this, Vincent,

It should be clear from public-test-infra that I am optimistic about in-browser testing APIs, and uncertain about depending on WebDriver, at least for the big APIs like Bluetooth and USB. I would love pushback from Chromium folks that disagree. However, even if we all agree within Chromium, we get the value of shared tests only if they are also automated in some other engine.

So, I believe the main thing we can do to move forward is to collaborate with other vendors on the testing needs of specific APIs, to see if we are aligned on preferred solutions. From Giovanni's list, are there any that seem like extra good candidates, that are currently being implemented in other engines?

I would be particularly interested to hear if there are any APIs with fairly modest testing needs, where the API surface if implemented in WebDriver would only amount to a new command or two. Although I'd love a simple recommendation for all new APIs, it might not be a one-size-fits all, so I would like to go through the exercise to see what an hypothetical (small) WebDriver extension would look like. Any candidates?

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAEmk%3DMZor7j7NSAd%2B%3DukrSnmw5peg6qpLcAoWGRtDUuHVVVbgA%40mail.gmail.com.

[Domenic Denicola]

My main concern with not using WebDriver is that web developers also need to be able to test these APIs. Will we be providing special browser builds with the testing APIs enabled, so that they can write automated tests against their web apps simulating the presence and behavior of certain Bluetooth devices? That’s what WebDriver would give us (in addition to fitting in to the framework web developers are already using for automated testing of their sites).

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYf8t92uwby6gka%3D1o9t635vYN1fpk2hajpe_%2Bpx3kAgNQ%40mail.gmail.com.

[Giovanni Ortuño]

+pfeldman who in the past argued against using WebDriver for the Bluetooth Test API.

Thanks for your replies Domenic and Philip.

On Fri, Apr 7, 2017 at 1:31 PM Domenic Denicola <d...@domenic.me> wrote:

My main concern with not using WebDriver is that web developers also need to be able to test these APIs. Will we be providing special browser builds with the testing APIs enabled, so that they can write automated tests against their web apps simulating the presence and behavior of certain Bluetooth devices? That’s what WebDriver would give us (in addition to fitting in to the framework web developers are already using for automated testing of their sites).

The proposal for exposing web-platform testing APIs rather than doing it through Web Driver does not rule out the possibility of exposing them to developers interested in using them.

The test APIs we currently use in our LayoutTests are implemented by overriding Mojo Interfaces and Services from Javascript. Currently this can only be done when running Layout Tests but there are no significant technical obstacles that would prevent us from exposing this capability behind a flag. Of course such a thing would have to go through security review.

Besides the small benefit of having many tests written this way already, this approach results in no binary size increase since all the code that implements these Test APIs can be loaded together with the tests. I'm currently looking at how Bluetooth's test API would be implemented using Web Driver to estimate the binary size impact but my guess is that it will be just as expensive as a C++ Mojo Interface: 100KB.

 

From: Philip Jägenstedt [mailto:foo...@chromium.org]
Sent: Friday, April 7, 2017 02:28
To: Reilly Grant <rei...@chromium.org>; Vincent Scheib <sch...@chromium.org>; platform-architecture-dev <platform-arc...@chromium.org>; platform-pr...@chromium.org; Giovanni Ortuño <ort...@chromium.org>; Matt Giuca <mgi...@chromium.org>; Mike Pennisi <mi...@bocoup.com>; James Graham <ja...@hoppipolla.co.uk>
Subject: Re: Chromium Architecture for exposing web-platform testing APIs

 

Thanks for pushing on this, Vincent,

 

It should be clear from public-test-infra that I am optimistic about in-browser testing APIs, and uncertain about depending on WebDriver, at least for the big APIs like Bluetooth and USB. I would love pushback from Chromium folks that disagree. However, even if we all agree within Chromium, we get the value of shared tests only if they are also automated in some other engine.

 

So, I believe the main thing we can do to move forward is to collaborate with other vendors on the testing needs of specific APIs, to see if we are aligned on preferred solutions. From Giovanni's list, are there any that seem like extra good candidates, that are currently being implemented in other engines?

 

I would be particularly interested to hear if there are any APIs with fairly modest testing needs, where the API surface if implemented in WebDriver would only amount to a new command or two. Although I'd love a simple recommendation for all new APIs, it might not be a one-size-fits all, so I would like to go through the exercise to see what an hypothetical (small) WebDriver extension would look like. Any candidates?

 

Two APIs come to mind: Geolocation and WebShare. Geolocation's test API should be very simple but the API requires a permission. WebShare is somewhat new; it would also have a simple Test API and doesn't require permission.

[Matt Giuca]

On Fri, 7 Apr 2017 at 14:34 Giovanni Ortuño <ort...@chromium.org> wrote:

+pfeldman who in the past argued against using WebDriver for the Bluetooth Test API.

Thanks for your replies Domenic and Philip.

On Fri, Apr 7, 2017 at 1:31 PM Domenic Denicola <d...@domenic.me> wrote:

My main concern with not using WebDriver is that web developers also need to be able to test these APIs. Will we be providing special browser builds with the testing APIs enabled, so that they can write automated tests against their web apps simulating the presence and behavior of certain Bluetooth devices? That’s what WebDriver would give us (in addition to fitting in to the framework web developers are already using for automated testing of their sites).

The proposal for exposing web-platform testing APIs rather than doing it through Web Driver does not rule out the possibility of exposing them to developers interested in using them.

The test APIs we currently use in our LayoutTests are implemented by overriding Mojo Interfaces and Services from Javascript. Currently this can only be done when running Layout Tests but there are no significant technical obstacles that would prevent us from exposing this capability behind a flag. Of course such a thing would have to go through security review.

Besides the small benefit of having many tests written this way already, this approach results in no binary size increase since all the code that implements these Test APIs can be loaded together with the tests. I'm currently looking at how Bluetooth's test API would be implemented using Web Driver to estimate the binary size impact but my guess is that it will be just as expensive as a C++ Mojo Interface: 100KB.

 

From: Philip Jägenstedt [mailto:foo...@chromium.org]
Sent: Friday, April 7, 2017 02:28
To: Reilly Grant <rei...@chromium.org>; Vincent Scheib <sch...@chromium.org>; platform-architecture-dev <platform-arc...@chromium.org>; platform-pr...@chromium.org; Giovanni Ortuño <ort...@chromium.org>; Matt Giuca <mgi...@chromium.org>; Mike Pennisi <mi...@bocoup.com>; James Graham <ja...@hoppipolla.co.uk>
Subject: Re: Chromium Architecture for exposing web-platform testing APIs

 

Thanks for pushing on this, Vincent,

 

It should be clear from public-test-infra that I am optimistic about in-browser testing APIs, and uncertain about depending on WebDriver, at least for the big APIs like Bluetooth and USB. I would love pushback from Chromium folks that disagree. However, even if we all agree within Chromium, we get the value of shared tests only if they are also automated in some other engine.

 

So, I believe the main thing we can do to move forward is to collaborate with other vendors on the testing needs of specific APIs, to see if we are aligned on preferred solutions. From Giovanni's list, are there any that seem like extra good candidates, that are currently being implemented in other engines?

 

I would be particularly interested to hear if there are any APIs with fairly modest testing needs, where the API surface if implemented in WebDriver would only amount to a new command or two. Although I'd love a simple recommendation for all new APIs, it might not be a one-size-fits all, so I would like to go through the exercise to see what an hypothetical (small) WebDriver extension would look like. Any candidates?

 

Two APIs come to mind: Geolocation and WebShare. Geolocation's test API should be very simple but the API requires a permission. WebShare is somewhat new; it would also have a simple Test API and doesn't require permission.

I'm the TL for Web Share. I would be happy to help change our tests over to this format. Our surface is very small and it fits the bill for this feature (requires user gesture and shows UI which needs to be mocked by the test harness). My only concern is that Web Share isn't standardized and only exists in Chrome, so it isn't a good test case for cross-browser test harnessing.

 

You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAMDKGBVQeUcJxw_kVbyjxeJeYFoCTNDJBPMuRVBLp9Rq9ssvAA%40mail.gmail.com.

[Vincent Scheib]

If using Web Driver was free to ship to stable channel, I'd be tempted to increase the scope of our goals beyond web platform testing for multiple browsers to also include facilitating testing for web developers. But, there is a cost in at least binary size for billions of users who will never need that, and technical complexity as well. Web developers can test on stable chrome by overriding our APIs with fake API implementations. And, no matter how we implement web platform test apis in content shell they have the option to use that as well. So, I don't see "but, web developers could test their apps if we use Web Driver" as a valid argument.

[Pavel Feldman]

The common baseline for browser automation and testing is Chrome's "Remote Debugging Protocol". ChromeDriver is implemented on top of it as well as every other automation and testing tool that it outside browser. Our own perf bots and Telemetry are using it, hundreds of third party projects use it.

The issue that Giovanni is referring to is that bluetooth (and some other domains with heavy state) are sometimes expensive to expose using remote debugging. Every connection, every device would need to have an ID that serves as a handle and allows operations on the entity. If API surface is huge, it renders protocol inefficient. Having said that, we can revisit it and expose. It needs to be exposed for WebDriver to be able to use it, previously WebDriver injected its own extensions with privileged APIs to perform tests, but this is now deprecated and it is eliminating remains of it.

Pavel

[Philip Jägenstedt]

On Fri, Apr 7, 2017 at 11:39 AM Matt Giuca <mgi...@chromium.org> wrote:

On Fri, 7 Apr 2017 at 14:34 Giovanni Ortuño <ort...@chromium.org> wrote:

I would be particularly interested to hear if there are any APIs with fairly modest testing needs, where the API surface if implemented in WebDriver would only amount to a new command or two. Although I'd love a simple recommendation for all new APIs, it might not be a one-size-fits all, so I would like to go through the exercise to see what an hypothetical (small) WebDriver extension would look like. Any candidates?

 

Two APIs come to mind: Geolocation and WebShare. Geolocation's test API should be very simple but the API requires a permission. WebShare is somewhat new; it would also have a simple Test API and doesn't require permission.

I'm the TL for Web Share. I would be happy to help change our tests over to this format. Our surface is very small and it fits the bill for this feature (requires user gesture and shows UI which needs to be mocked by the test harness). My only concern is that Web Share isn't standardized and only exists in Chrome, so it isn't a good test case for cross-browser test harnessing.

I found the proto-spec at https://github.com/WICG/web-share/blob/master/docs/interface.md, is any other vendor currently giving any feedback on this?

Even if they aren't, this might be a small and contained API to get some experience with test automation. Do you have a team list I could mail to start that discussion?

[Philip Jägenstedt]

Thanks Pavel, that's very helpful. I've tried once before to trace my way from WebDriver to inside Blink for the take screenshot command. For my education, could you point out which bits of this are in the Remote Debugging Protocol, and what the layers immediately above and below are?

My takeaway from this discussion is that for APIs with a large surface, we (Chromium/Blink) would prefer to pursue in-browser testing APIs, in a way that leaves the door open to exposing them in end-user Chrome builds. (This means that if we want no binary size increase at all, some bits would have to been downloaded on demand, which isn't something I'm familiar with.)

For APIs with a smaller surface area, we need to explore a bit. In any case, I don't think we can make a general recommendation about how to test new features yet.

I'll follow up on public-test-infra now.