Writing web-platform-tests using non-web-exposed APIs is now easier

[James Graham]

Two changes that recently landed in web-platform-tests make it easier to
write tests that require access to non-web-exposed features:

* testdriver APIs now work in many situations involving multiple
browsing contexts origins

* SpecialPowers is now available in gecko-only web-platform-tests

testdriver
==========

testdriver [1] is a cross-browser API provided by wpt for performing
privileged actions, notably those involving trusted user gestures.
Because it's cross-browser this is the preferred way to write tests
requiring such interactions, where possible.

The default implementation of testdriver uses WebDriver on the backend,
and where specs provide WebDriver APIs for automation (e.g. [2]) they
can be exposed to wpt via testdriver.

Previously testdriver APIs only worked in the browsing context
containing the test harness. Now it's possible to use them in any
browsing context that is reachable from the context containing the
harness (i.e. any context that can postMessage to the test window). Full
documentation is given in [1].

Unfortunately handling cases where the contexts can't communicate e.g.
rel=noopener remains challenging because of limitations in webdriver/the
way we communicate between the harness and the browser. It isn't
impossible to fix these cases, but will require additional work. Please
let me know when you're blocked writing cross-browser tests because of
such limitations since this will help prioritise the work.

SpecialPowers
=============

The SpecialPowers API that's commonly used in gecko-specific tests in
other harnesses is now available for use in gecko-specific
web-platform-tests i.e. those that live in
testing/web-platform/mozilla/tests These tests obviously can't be run
cross-browser and aren't upstreamed. Therefore it's preferable to write
a shareable test using testdriver or similar where possible. However
there are some cases where that simply doesn't provide the required
features and rather than forcing authors to write mochitest-plain tests
for those cases it's now (hopefully) possible to do everything required
for content tests in wpt.

Currently this is only enabled for js (i.e. testharness) tests. If there
are use cases that require SpecialPowers in other test types (e.g. wpt
reftests) please let me know.

To be really clear here, this isn't suitable for anything that would
have previously used e.g. a browser-chrome test. The intent is to make
it possible to test web apis in a way that requires occasional access to
internals, not to allow testing the internals themselves.

Future Work
===========

We want to keep making this kind of improvement to wpt, especially where
doing so allows us to drive compatibility improvements by writing
cross-browser tests where it was previously impossible.

We are proposing and making an initial implementation of the Browser
Testing API spec [3], which is designed to provide a place to specify
test-only features that don't make sense in WebDriver; the initial scope
is a function to invoke a garbage collection; this has been a
longstanding feature request. If there are additional APIs that people
think would make sense in such a spec, please let me know.

In general if there are places where wpt doesn't meet our requirements
for testing, or there are chances to improve the test writing ergonomics
please let me know and we can make sure that work gets appropriate
priority. Shared tests remains a key technique for ensuring
interoperability between different browser engines.

[1] http://web-platform-tests.org/writing-tests/testdriver.html
[2] https://w3c.github.io/permissions/#automation
[3] https://jgraham.github.io/browser-test/

[Mirko Brodesser]

On Wednesday, October 21, 2020 at 11:56:45 AM UTC+2, James Graham wrote:
> Two changes that recently landed in web-platform-tests make it easier to
> write tests that require access to non-web-exposed features:
>
> * testdriver APIs now work in many situations involving multiple
> browsing contexts origins
>
> * SpecialPowers is now available in gecko-only web-platform-tests
>

Thanks!

Supporting synthesizing drag-and-drop events [1] and convenience functions for checking the clipboard [2] would be helpful. In general, it's presumably worth checking what kind of convenience functions are offered [3] and used by mochitests.

[1] https://searchfox.org/mozilla-central/rev/e1d1f043957191616721b9e8bf811c0aab8a203a/testing/mochitest/tests/SimpleTest/EventUtils.js#4-26.
[2] https://searchfox.org/mozilla-central/rev/e1d1f043957191616721b9e8bf811c0aab8a203a/testing/mochitest/tests/SimpleTest/SimpleTest.js#1224
[3] https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/

[James Graham]

On 26/10/2020 14:11, Mirko Brodesser wrote:

> Supporting synthesizing drag-and-drop events [1]

That seems like the kind of thing that ought to be covered by
testdriver. Would something like
new test_driver.Actions()
.pointerMove(0,0,{origin:elem1})
.pointerDown()
.pointerMove(0,0,{origin:elem2})
.pointerUp()
.send();

work?

Obviously if the concern is verbosity we could add helper functions for
common patterns. Looking at the other functions there, I know wheel
events are missing; that's a recent addition to the webdriver spec that
we haven't implemented in marionette, but we can prioritise that work if
it's needed for tests.

> and convenience functions for checking the clipboard [2] would be helpful. In general, it's presumably worth checking what kind of convenience functions are offered [3] and used by mochitests.

Clipboard is a good idea.

Looking at SpecialPowers usage [1] reveals a lot that's just poking at
other internal APIs, so it's hard to see what the underlying
requirements are.

EventUtils usage [2] looks like it's mostly stuff that should be covered
in testdriver, with the exception of IME stuff which might be
interesting to investigate.

[1] https://paste.mozilla.org/stbR88is
[2] https://paste.mozilla.org/g4qMtnMx