Re: Chromium and test automation

[Stephen Mcgruer]

(+ecosystem-infra@, after confirming with Raphael that they are ok with adding a mailing list)

Hey Raphael,

I think you describe the situation perfectly, and I'm not sure I have many additional findings that would be useful. I have a never-finished WebDriver + WPT: State of the Union document from the middle of this year, but it does little more than list a few pain-points with the webdriver situation.

I agree that commiting to a WebDriver API whilst developing specs is a heavy burden, though I note that technically any 'proper' use of MojoJS in the context of WPT also means committing to spec changes (since one is meant to define a Test-Only API). But of course it is easier to write Test-Only APIs than WebDriver Extensions for most folks.

It doesn't seem the worst thing for a spec to have 'non-interoperable' tests whilst it is still in (a) heavy development and (b) only has one vendor interested in implementing it. When either of those change (e.g. the spec is maturing or another vendor wants to join in), that's where defining an interoperable testing story matters. Perhaps we should have a setup where we say "whilst you're still doing early iteration, it's fine to use testdriver-vendor.js with internal Chromium APIs/whatever" to keep the bar as low as possible for adding your tests, and then at a certain point say "ok, now you have to define interoperable test APIs" (which would preferably be webdriver extensions).

I'd be interested to understand what 'stage' you feel the APIs your team is working on (Screen Wake Lock, Generic Sensors, Image Capture, NFC, etc) are at, and how you feel that interacts with the interoperability of your testing strategies.

Thanks,

Stephen

On Wed, 21 Oct 2020 at 06:49, Kubo Da Costa, Raphael <raphael.ku...@intel.com> wrote:

Hi Stephen,

At Monday's TPAC meeting you mentioned some work at Google to sort out the test automation story in Chromium. That's something I've worked on in a way or another for a while now (I presented some of what we were doing with the Generic Sensors APIs at the 2018 TPAC), but since we didn't have a lot of time and focusing too much on Chromium is generally frowned upon in those meetings I figured it'd be a better idea to reach out to you separately :-)

As far as I can see, there are 3 options for people implementing specs in Blink:

1. Add WebDriver extensions to the spec and use them in tests (e.g. the Reporting API spec)

2. Spec test-only APIs and use them in tests (e.g. WebUSB)

3. Implement some mock APIs and (optionally) document them in a README (e.g. many of the files in resources/chromium in WPT).

While 1) is the preferred option from an interop point of view, it's definitely a lot more work -- even though we've got an Automation section in the Generic Sensors spec, we still haven't got around to actually implementing it and still rely on MojoJS and mocks... For teams who do not normally interact with the WPT or WebDriver communities, I guess the incentive is even smaller, as one needs to be familiar with the ChromeDriver code base, know their way from chrome/test/chromedriver back to Blink and then prototype and discuss a testdriver.js implementation. For specs under heavy development, I believe committing to a certain WebDriver API on top of it all makes it extra hard to prefer this alternative to just using MojoInterfaceInterceptor, touching a single repository and only worrying about the Blink part of the spec implementation.

Having the cases in 3) move to 2) would definitely be an improvement, but I guess it's still not a move in the direction we'd generally prefer in WPT-land.

Anyway, I'm interested in your findings as well, and would like to help where possible :-) My team works on APIs like Screen Wake Lock, Generic Sensors, Image Capture, NFC (to a certain extent) and a few others, many of which rely on undocumented mock interfaces and could be part of any experiments you guys at Google might be considering in order to address this issue.

Cheers,

Raphael

[Raphael Kubo Da Costa]

Hi Stephen,

Thanks for the reply; I'm sending a short one right now just to let you know I'll get back to this next week as the TPAC activity slows down.