Intent to Ship: Generic Sensors-based Motion Sensors

[alexander...@intel.com]

Contact emails

alexander...@intel.com, mikhail.p...@intel.com, anssi.ko...@intel.com

Explainer

https://w3c.github.io/motion-sensors/

https://developers.google.com/web/updates/2017/09/sensors-for-the-web

Spec

https://w3c.github.io/sensors/

https://w3c.github.io/accelerometer/

https://w3c.github.io/gyroscope/

https://w3c.github.io/orientation-sensor/

(Published as W3C Candidate Recommendations on 20 March 2018.)

Tag reviews:

https://github.com/w3ctag/design-reviews/issues/110

https://github.com/w3ctag/design-reviews/issues/207

Summary

The Generic Sensor API is a framework for adding sensor devices to the web platform in a consistent way.

This intent includes shipping the following interfaces: Accelerometer, LinearAccelerationSensor, Gyroscope, AbsoluteOrientationSensor and RelativeOrientationSensor. These are extensible replacements for the DeviceMotionEvent and DeviceOrientationEvent interfaces and provide no additional data. The existing 60 Hz sampling rate limit is still in effect.

We believe the modular approach of these APIs provides needed flexibility to implementers to consider the security and privacy implications of sensors and apply policy and UX changes in a way that is transparent to developers. As we continue to consider these implications we will follow up with additional intents for other sensor types.

Link to “Intent to Implement” blink-dev discussion

Intent to Implement: Generic Sensor & Ambient Light Sensor APIs

Intent to implement Accelerometer, Gyroscope and Magnetometer Sensor APIs

Link to Origin Trial feedback summary

An origin trial was conducted however usage did not extend beyond development by the team working on the specification and implementation. The Intent to Experiment thread.

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Demo link

https://intel.github.io/generic-sensor-demos/

Debuggability

DevTools currently only supports setting a device orientation that will be reflected by the DeviceOrientationEvent interface. This should be extended with support for the Generic Sensors sensor types as well as to provide consistency with the legacy APIs.

Tracking bug: https://crbug.com/816478

Risks

Interoperability and Compatibility

As a new API the main interoperability and compatibility risk of shipping is that no other implementers do the same. Though the TAG review process we have strong signals that this is the right direction in which to move for providing sensor interfaces on the web.

Edge: No signals, Edge claims to be shipping a preview of the AmbientLightSensor object which is based on the same Generic Sensor API framework however this appears to be incorrect. The status page also lists Chrome and Opera as having shipped this API which is incorrect.

Firefox: Mixed signals, developers have shown positive signals about Generic Sensors as a better API for sensors however they have shown concern about the privacy impact of sensor APIs in general. Discussion of Mozilla’s position is underway and a meta-bug has been filed.

Safari: No signals

Web developers: Positive, the webvr-polyfill has been patched to use Generic Sensors when available and the feedback from the developers is positive. The webvr-polyfill v0.10.3 release notes recommend users to enable Generic Sensors for better performance.

While not the web platform, Fitbit is using this API in their JavaScript-based development environment for Fitbit OS, which increases our confidence in the design.

Ergonomics

This motion sensor subset of the APIs based on the Generic Sensors API provides improved developer ergonomics over the existing DeviceMotionEvent and DeviceOrientationEvent objects. The new sensor objects are configurable with a sampling frequency (up to the 60 Hz default provided by the legacy API) and the option of either device or screen-relative coordinates. The sensor object provides separate events for activation, updates and errors. For WebGL applications device orientation is provided in either matrix or quaternion form.

These enhancements also provide more points of flexibility for implementers to apply policy such as permissions (an “access denied” error can be signaled through an event) and limits on sampling rate (the requested value is only an upper bound).

Activation

Developers can use this new API through a polyfill written by Kenneth Christiansen that builds basic support for these APIs on top of the legacy sensor interface: https://github.com/kenchris/sensor-polyfills/

Is this feature fully tested by web-platform-tests? Link to test suite results from wpt.fyi.

There are basic IDL tests in web-platform-tests however due to the need for an interface to inject test data the majority of the tests are still in LayoutTests. These could be moved to web-platform-tests if the mock Mojo sensor provider interface implementation were moved into a Chromium-specific support library as has been done for the Web Bluetooth and WebUSB tests.

WPT results: https://wpt.fyi/generic-sensor

Upstreaming issue: https://crbug.com/816462 https://github.com/w3c/web-platform-tests/issues/9686

Entry on the feature dashboard

https://www.chromestatus.com/feature/5698781827825664

[Mikhail Pozdnyakov]

Any updates on the intent?

Thanks,
Mikhail

[Philip Jägenstedt]

Let me say up front that I think the new sensor APIs make a lot of sense, and the length of this reply is all in good faith, some due diligence.

This is shipping a rather large new API surface (the first set of Generic Sensors-based APIs) and I think the main thing to discuss here is the interop risk and privacy, but let me start with my favorite thing, testing:

Although only very surface-level tests can be written in WPT becuse of the issue you linked, it would be nice to see the results of these tests:

https://wpt.fyi/generic-sensor

https://wpt.fyi/accelerometer

https://wpt.fyi/gyroscope

https://wpt.fyi/orientation-sensor

I'm sorry to say this is much harder than it should be. We don't run Chrome Dev on wpt.fyi and w3c-test.org doesn't quite work. So I wrote some documentation on how to use wpt run for this and tried:

./wpt run --binary `which google-chrome-unstable` --binary-arg=--enable-blink-features=Sensor chrome generic-sensor/ accelerometer/ gyroscope/ orientation-sensor/

There are fairly many failures, and happily they seem to match the *-expected.txt in LayoutTests/external/wpt. One thing that stands out are many harness errors because GravitySensor is not defined. That's in the spec, so I wonder if this omission is intentional?

Whether intentional or not, can you look into fixing the harness errors and any other unexpected test failures? This would be very helpful to increase confidence that the implementation matches the spec, and that the tests will be useful for the next implementer.

Interop risk and privacy, then, which are related.

Since we also don't have any test results from Edge Insider Preview I tried it on a laptop with experimental features enabled in about:flags and AFAICT there aren't any APIs with "Sensor" in the name...

+Travis Leithead, do you know if there's any Generic Sensors API work in Edge that has reached the public yet? Should we take your status page as support of this set of APIs even if it's not possible to test yet?

Has there been security and/or privacy review of these new APIs, or do they expose exactly the same information as existing APIs with the same granularity and (lack of) permissions, so that this is deemed not necessary?

I've commented on the Mozilla discussion and it looks to me like the API would deal well with either the absence or presence of extra permissions for certain sensors. Some of the same ground was covered in the TAG reviews. Is there history/discussion here that we should be aware of?

How about WebKit, has there been any movement on these APIs there?

Thank you all for your work on this, and your patience!

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/9911f5f3-5d29-42a8-af59-281a54a3d32d%40chromium.org.

[Mikhail Pozdnyakov]

Hi Philip,

Thank you for your reply!

Let me try to address some of your concerns.

Regarding testing, the intended APIs have already rather good layout tests coverage https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/sensor/ and now we are actively working on enabling WPT tests in Chromium, the dedicated CL https://chromium-review.googlesource.com/c/chromium/src/+/980886 is expected land soon.

The GravitySensor interface is not part of this intent (and it is not yet implemented) because it exposes something different from what is being exposed by the DeviceOrientation

and the DeviceMotion APIs. And the goal of this intent is to provide "extensible replacements for the DeviceMotionEvent and DeviceOrientationEvent interfaces and provide no additional data."

The privacy and security risks had been analyzed during the W3C wide review (https://github.com/w3c/sensors/issues/299) and the review feedback had been addressed before the APIs specifications were published as W3C CRs. Broadly speaking, the spec CR publications signaled W3C community has completed its wide review and considers the specs shippable.

BR,

Mikhail

[alexander...@intel.com]

Hi Philip,

> Since we also don't have any test results from Edge Insider Preview I tried it on a laptop with experimental features enabled in about:flags and AFAICT there aren't any APIs with "Sensor" in the name…

As indicated in interop section, there are no signals from Edge, however, platform status erroneously states that AmbientLightSensor is shipped in Edge preview build 10586+.

>Has there been security and/or privacy review of these new APIs, or do they expose exactly the same information as existing APIs with the same granularity and (lack of) permissions, so that this is deemed not necessary?

Security and privacy issues that have been raised during first and second wide review rounds, were addressed. Also, changes related to issues created by PING IG, were incorporated to the specification. Summary of the changes can be found here: https://github.com/w3c/sensors/issues/299#issuecomment-369924348

Information about security and privacy improvements over DeviceMotion and DeviceOrientation event API can be found in Security and Privacy section.

>I've commented on the Mozilla discussion and it looks to me like the API would deal well with either the absence or presence of extra permissions for certain sensors.

Concrete sensors define set of permissions, and in order to expose sensor readings, UA must obtain permission for each token. How permission is obtained by the UA is outside of scope of the specification.

Specification improvements for user consenting were addressed in https://github.com/w3c/sensors/issues/352#issuecomment-370767932.

> How about WebKit, has there been any movement on these APIs there?

There are no signals, as indicated in the original Intent to ship mail.

Best regards,

Alex

On Thursday, March 29, 2018 at 2:56:43 PM UTC+3, Philip Jägenstedt wrote:

[Travis Leithead]

+Travis Leithead, do you know if there's any Generic Sensors API work in Edge that has reached the public yet? Should we take your status page as support of this set of APIs even if it's not possible to test yet?

 To my knowledge, we don't have any pending Generic Sensor work in-flight. We have been monitoring the Devices and Sensor WG, and are fairly happy with the architecture that has been developed, so you can take that as support for the various sensor features--even if we don't have any plans to ship them in the near term. We would expect to take full advantage of the Security and Privacy mitigation strategies outlined in the spec: https://www.w3.org/TR/generic-sensor/#mitigation-strategies-case-by-case)

The AmbientLightSensor support (mentioned here: https://developer.microsoft.com/en-us/microsoft-edge/platform/status/ambientlightsensorapi/) is based on the old Ambient Light spec (documented on MDN here: https://developer.mozilla.org/en-US/docs/Web/API/Ambient_Light_Events). We have the variety for Device*Event objects implemented--none of which are based on Generic Sensor API.

[Ojan Vafai]

LGTM1 once Philip's questions are resolved.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2d0c2255-45b1-49d5-933f-76975f6774f7%40chromium.org.

[Philip Jägenstedt]

On Thu, Mar 29, 2018 at 4:19 PM Mikhail Pozdnyakov <mikhail.p...@intel.com> wrote:

Hi Philip,

Thank you for your reply!

Let me try to address some of your concerns.

Regarding testing, the intended APIs have already rather good layout tests coverage https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/sensor/ and now we are actively working on enabling WPT tests in Chromium, the dedicated CL https://chromium-review.googlesource.com/c/chromium/src/+/980886 is expected land soon.

Thanks, I've commented there.

 

The GravitySensor interface is not part of this intent (and it is not yet implemented) because it exposes something different from what is being exposed by the DeviceOrientation

and the DeviceMotion APIs. And the goal of this intent is to provide "extensible replacements for the DeviceMotionEvent and DeviceOrientationEvent interfaces and provide no additional data."

That's what I thought, just checking because it is in https://w3c.github.io/accelerometer/ and in the tests.

 

The privacy and security risks had been analyzed during the W3C wide review (https://github.com/w3c/sensors/issues/299) and the review feedback had been addressed before the APIs specifications were published as W3C CRs. Broadly speaking, the spec CR publications signaled W3C community has completed its wide review and considers the specs shippable.

Sorry, I wasn't clear, I was wondering whether any Chrome-internal review has happened. I've poked +Mike West and defer to whatever he has to say.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/ba0ac946-7cb3-41f9-9321-43207ecd6c2b%40chromium.org.

[Philip Jägenstedt]

Thanks for clarifying, Travis, mystery solved.

[Philip Jägenstedt]

LGTM2, I've deferred to +Mike West on whether there has been / should be Chrome-internal security review, and on testing let's discuss on the review. I would like to see that done before flipping the flag to ship, but if the branch point gets close let's discuss, also on the review.

As an aside, I wonder if https://github.com/kenchris/sensor-polyfills is what we should recommend web developers to use? It seems quite important to fall back to the existing APIs for a long time to come, and it should be easy to do so.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANMdWTsiO%2Bs7EXVEXixc0mEoigxsnQN5vxmY2bnCCAZdaxBcyA%40mail.gmail.com.

[Reilly Grant]

On Tue, Apr 3, 2018 at 5:16 AM Philip Jägenstedt <foo...@chromium.org> wrote:

LGTM2, I've deferred to +Mike West on whether there has been / should be Chrome-internal security review, and on testing let's discuss on the review. I would like to see that done before flipping the flag to ship, but if the branch point gets close let's discuss, also on the review.

For context, the security one-pager at go/generic-sensors-security covers the results of our previous conversations with Chrome-internal security reviewers. In the language of that document this is the "phase 1" launch where no new sensor types are introduced. In parallel we are working on "phase 2" which implements new Sensor API Permissions UX. This will enable "phase 3" which is introducing new sensor types for which we think a permission is required.

 

As an aside, I wonder if https://github.com/kenchris/sensor-polyfills is what we should recommend web developers to use? It seems quite important to fall back to the existing APIs for a long time to come, and it should be easy to do so.

On Tue, Apr 3, 2018 at 1:09 AM Ojan Vafai <oj...@chromium.org> wrote:

LGTM1 once Philip's questions are resolved.

On Fri, Mar 30, 2018 at 12:20 PM Travis Leithead <travis....@gmail.com> wrote:

+Travis Leithead, do you know if there's any Generic Sensors API work in Edge that has reached the public yet? Should we take your status page as support of this set of APIs even if it's not possible to test yet?

 To my knowledge, we don't have any pending Generic Sensor work in-flight. We have been monitoring the Devices and Sensor WG, and are fairly happy with the architecture that has been developed, so you can take that as support for the various sensor features--even if we don't have any plans to ship them in the near term. We would expect to take full advantage of the Security and Privacy mitigation strategies outlined in the spec: https://www.w3.org/TR/generic-sensor/#mitigation-strategies-case-by-case)

The AmbientLightSensor support (mentioned here: https://developer.microsoft.com/en-us/microsoft-edge/platform/status/ambientlightsensorapi/) is based on the old Ambient Light spec (documented on MDN here: https://developer.mozilla.org/en-US/docs/Web/API/Ambient_Light_Events). We have the variety for Device*Event objects implemented--none of which are based on Generic Sensor API.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2d0c2255-45b1-49d5-933f-76975f6774f7%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CANMdWTsiO%2Bs7EXVEXixc0mEoigxsnQN5vxmY2bnCCAZdaxBcyA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYcCHDZnq3VCwweq8Y9gpRWaiaO3xCJn6Ae3cZOwu5NqeA%40mail.gmail.com.

[Chris Harrelson]

LGTM3

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEmk%3DMb5zsmgDtpr01gpp99xqWPOBhoEyUHE5LFPNOiv3qQvag%40mail.gmail.com.