Intent to Prototype and Ship: Generic Sensor WebDriver endpoints

[Kubo Da Costa, Raphael]

I'm sending this Intent after PSA: WebDriver changes now go through the API owners intent process. This change does not expose new user-facing API or change existing user-facing behavior. It does add new CDP and ChromeDriver endpoints that are used in WPT.

The contents of some fields here were based on PSA: Adding WebDriver commands for interacting with FedCM dialogs.

Contact emails

raphael.ku...@intel.com, juha.j...@intel.com

Specification

https://github.com/w3c/sensors/pull/470

Design doc

https://docs.google.com/document/d/1JS2Wnyc9GiS_u1Ry3w4KsCyd51g8fm4RDEUcanZgzbM/edit?usp=sharing

Summary

This exposes WebDriver commands for creating, removing, querying and setting readings for so-called virtual sensors: sensors that do not depend on underlying hardware or operating system support and can be used for testing.

Blink component

Blink>Sensor

Motivation

The Generic Sensor API specification has had an Automation section since 2018. It defined several WebDriver extension commands for adding/removing/querying/updating what it called "mock sensors". The idea was the same: allow testing sensor APIs without depending on the availability of said sensors.

For a number of reasons (including some implementation-specific assumptions baked into the text), the previous version was never implemented, so testing the Generic Sensor APIs in WPT relied on Mojo interfaces being implemented as JS mocks, which made it pretty hard for other engines to run the same tests in case they wanted to implement this API.

We've rewritten the relevant section in the spec (including calling "mock sensors" "virtual sensors" now following WebAuthn's terminology), implemented the new version and converted the existing web tests to make use of them.

TAG review

None

TAG review status

Not applicable

Risks

Interoperability and Compatibility

This feature implements WebDriver extension commands in ChromeDriver that are used in web-platform-tests and can also be used by ChromeDriver users. This feature is therefore not web-exposed, but it does intend to help increase adoption of the spec by making it possible for any implementation to run the existing web tests without having to use JS mocks that are heavily Mojo-based although not dependent on Mojo.

Gecko: No signal Gecko does not implement the Generic Sensor APIs.

WebKit: No signal WebKit does not implement the Generic Sensor APIs.

Web developers: No signals

Other signals: This allows closing https://github.com/web-platform-tests/wpt/issues/9686

Ergonomics

n/a

Activation

The new WebDriver extension commands are exposed in ChromeDriver's Python library. Future work includes exposing them to Selenium, Playwright and/or Puppeteer.

Furthermore, while not precisely an activation risk, as explained in the design doc using these features together with the existing DeviceOrientation CDP domain (optionally with DevTools frontend implementation) can lead to some confusion due to the way the Device Orientation spec is implemented in Chromium (it is based on the Generic Sensor code under the hood). If the DeviceOrientation.setDeviceOrientationOverride() CDP command is called, some virtual sensor types will not have an effect in Blink; conversely, if some virtual sensor types are created without the aforementioned CDP command being invoked, they can be used to control the Device Orientation/Motion implementation in Blink.

Security

n/a, this is exposed through WebDriver. There have been changes to the Generic Sensor code in //content and //services, described in the design doc and under review by reillyg@.

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

This is a debugging feature. It exposes new ChromeDriver and CDP endpoints, but the DevTools frontend has not been touched.

Is this feature fully tested by web-platform-tests?

Yes

Requires code in //chrome?

True: //chrome/test/chromedriver

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1278377

Estimated milestones

Shipping on desktop

118

Shipping on Android

118

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5173284500406272

This intent message was generated by Chrome Platform Status.

[Mathias Bynens]

ChromeDriver owner here. Thank you for applying the new process!

This LGTM, with just a few questions. I know Gecko and WebKit don’t currently implement the Generic Sensor APIs. Question 1: Are they supportive of the Generic Sensor API? Question 2: Have they provided any signals on the proposed automation endpoint?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/DS0PR11MB73342BF79AC2F73824270FFCFE14A%40DS0PR11MB7334.namprd11.prod.outlook.com.

[Raphael Kubo da Costa]

Thanks, Mathias.

Mathias Bynens <mat...@chromium.org> writes:
> Question 1: Are they supportive of the Generic Sensor API?

Officially, no. It has been quite a while since we last asked them
though.

See the discussion with Mozilla here:
- https://github.com/mozilla/standards-positions/issues/35
and their corresponding standards position here:
- https://mozilla.github.io/standards-positions/#generic-sensor

WebKit didn't have a standards position tracker back then, but they did
comment on the Mozilla issue above:
- https://github.com/mozilla/standards-positions/issues/35#issuecomment-718365969

We've been trying to address the concerns in the Devices and Sensors WG
over the years and this will probably come up in some form at TPAC as
well.

With that said...

> Question 2: Have they provided any signals on the proposed automation
> endpoint?

I have not asked them given the above (and Apple is not part of the WG,
so it's also hard to ask for feedback on any specifics of the new text).
Personally, I'd like to do a few things before discussing this API with
WebKit and Mozilla again:
- Land these automation changes to remove the semi-dependency on Mojo
from the web tests so that it's a lot easier for other implementations
to test their code.
- Sort out any permission-related issues the spec might have (although
this is more of an implementation issue, with the Generic Sensors and
Device Orientation APIs in Blink not asking for permission by default
at the moment).
- Figure out if the Device Orientation spec (which all engines
implement) itself needs to be more tightly coupled with the Generic
Sensor API, at least for automation, for it to be testable in an
interoperable way.

[Mathias Bynens]

Thank you for the detailed explanation. Proceeding as you described SGTM.

[Mike Taylor]

LGTM1

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADizRgYkZ%3DH3qe8DKZdcm1bLqYnGCLFpyX%3DCBGCV%3DmTWG3EJZg%40mail.gmail.com.

[Yoav Weiss]

LGTM2 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/eb91049c-9d0f-44a2-bb4c-9987b12d34b0%40chromium.org.

[Chris Harrelson]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfV%2BZEtq%3DfezcPuE1GgMLD8eKo_XQQtVc7t9WgxvxcaLJg%40mail.gmail.com.