[blink-dev] Intent to implement: proximity sensor support.

[Tim Volodine]

Contact emails

timvo...@chromium.org

+ OWP team London TBD

Spec

http://www.w3.org/TR/proximity/

Summary

Implement proximity sensor support in Chrome.

Motivation

Specification is a candidate recommendation and already shipped in Firefox. There is an indication of significant usage by apps on Play Store and by other companies.

1. Turn screen off when close to ear. Potentially useful for WebRTC.

2. Air gestures / Hovering controls (Play Store apps [1]).

 - quick, hands are dirty, wearing gloves, in-car controls.

 - music/video controls: change tracks, start/stop.

 - accept/decline calls, slide show, turn off alarm, switch to voice control (e.g. Waze [2]).

 - moving items to different location, tap + air gesture.

3. Fun applications and future uses [1, 3].

 - adjusting sound volume, frequency, switch to speaker based on proximity.

 - proximity aware UI.

Compatibility Risk

Firefox: shipped.

Ongoing technical constraints

None.

Will this feature be supported on all five Blink platforms (Windows, Mac, Linux, ChromeOS, and Android)?

Yes (where possible).

OWP launch tracking bug?

Not yet.

Row on feature dashboard?

No.

Requesting approval to ship?

No. Initial implementation will be behind a flag.

References

(1) Play store data available offline.

(2) http://www.theverge.com/mobile/2012/2/9/2786594/waze-iphone-proximity-sensor-ios-update

(3) http://appleinsider.com/articles/14/02/27/apples-dynamic-user-interface-would-adapt-to-users-proximity-allowing-control-from-close-far

[Peter Beverloo]

I'm a bit worried that the use-cases you mention are purely theoretical use-cases, and would like to know more about the "significant usage" you found, which I'll ask about separately.

When I, as a developer, am faced with the decision of whether to build a native or a Web app, I'd start with listing the requirements of my app. Even when developing a Phone app, the ability to prevent accidental input isn't my top priority. I'd be more concerned with getting access to the person's contacts, keeping the device awake during the duration of the call, muting other audio playing on the device, maybe opening itself in the foreground if it isn't, and so on. Preventing accidental input could also be done by having focused UI, and requiring a more complicated gesture (e.g. sliding) to stop the conversation.

For air gestures, the granularity of the API often won't be high enough to effectively create gesture recognizers, unless the gestures are really basic -- slowly waving your hand three times, for example. The direction of a wave would not be detectable, either. Furthermore, it's noteworthy that all of these would require the device to be already unlocked with the page in the foreground, which makes most of the named use-cases unlikely.

I'd consider this feature to be a relatively weak nice to have, given that the impact will be very low. Wouldn't you consider features like ambient light events/getters and something like wake locks to be of higher priority? There are more important gaps to bridge.

To give some more technical feedback which may be relevant...

The specification defines two events, DeviceProximityEvent for indicating the distance to a nearby object, and UserProximityEvent for indicating the presence of a nearby object. This seems to be a security measure in case the user agent does not want to (or cannot) expose the precise distance between the device and a nearby object. The specification has a brief and very generic section on privacy considerations, but is inconclusive. Would we want to expose this to every website out there? Is there a need for a security mechanism?

It would be a lot better to have a single event (UserProximityEvent?) with an additional property for a distance estimate when both known and it's safe to expose.

For Android, it's important to note that the platform does not make any guarantees about the values returned by the proximity sensor. Many sensors on Android seem to be based on the amount of lux received by a front-facing sensor, and estimate the distance based on the relative darkness. Others return fixed values depending on whether an object is far or near. The specification seems to be crafted around this, and defines that whatever values the platform happens to offer should just be forwarded to the Web application. How interoperable will these values be with, say, a mobile Apple or Windows device?

If Blink were to support the Ambient Light API, a reasonable estimate of this functionality could be supported by a page monitoring the light conditions, and detecting when the amount of lux (steeply) decreases. When using WebRTC, a developer could approximate the distance of an object in front of the camera by measuring its velocity in relation to the device's orientation, but that can get complicated really quickly :-).

Thanks,

Peter

[Kostiainen, Anssi]

Hi Peter, Tim, All,

On 04 Mar 2014, at 01:20, Peter Beverloo <pe...@chromium.org> wrote:

[...]

> To give some more technical feedback which may be relevant...

Peter - thank you for your detailed technical feedback. Much appreciated.

> The specification defines two events, DeviceProximityEvent for indicating the distance to a nearby object, and UserProximityEvent for indicating the presence of a nearby object. This seems to be a security measure in case the user agent does not want to (or cannot) expose the precise distance between the device and a nearby object.

The main reason for the design is to allow better performing implementations, which should be of interest to Blink too. The UserProximityEvent will fire less frequently, so the implementations can optimize for that. Of course, as you mention, the partitioning could also be used as a security boundary by some implementations.

> The specification has a brief and very generic section on privacy considerations, but is inconclusive. Would we want to expose this to every website out there? Is there a need for a security mechanism?

The spec was reviewed by the W3C Privacy Interest Group (a group of privacy experts), and this section has been authored based on their feedback. The section given implementers recommendations, but does not mandate a specific approach to privacy and security to allow implementers to innovate and differentiate.

If we take Chrome as an example, I think the tab indicators used for capabilities such as sound, webcam and casting, might provide a nice UI hook to give an indication to the user a sensor is being used.

> It would be a lot better to have a single event (UserProximityEvent?) with an additional property for a distance estimate when both known and it's safe to expose.

Given an implementation of this feature is shipping in Firefox, I think we may not want to merge the interfaces at this time. If you feel strongly about this, I’d invite you to discuss this on the appropriate W3C mailing list (feel free to ping me off-the-list for more details).

> For Android, it's important to note that the platform does not make any guarantees about the values returned by the proximity sensor. Many sensors on Android seem to be based on the amount of lux received by a front-facing sensor, and estimate the distance based on the relative darkness. Others return fixed values depending on whether an object is far or near.

Good point. Some sensors will always be of better quality than others, so implementers should take that into consideration, especially if the platform APIs do not make any quarantees about the values.

> The specification seems to be crafted around this, and defines that whatever values the platform happens to offer should just be forwarded to the Web application. How interoperable will these values be with, say, a mobile Apple or Windows device?

Nothing in the spec is preventing an implementation from e.g. removing outliers before passing the value to the web content. Similarly, the frequency how often the events are fired is in control of the implementation. Some implementation may fire more frequently than others, and some will be more precise that others.

> If Blink were to support the Ambient Light API, a reasonable estimate of this functionality could be supported by a page monitoring the light conditions, and detecting when the amount of lux (steeply) decreases. When using WebRTC, a developer could approximate the distance of an object in front of the camera by measuring its velocity in relation to the device's orientation, but that can get complicated really quickly :-).

Well, you just seem to have described how someone could try to polyfill this ;-) I think we agree such a polyfill would not be very reliable in practice, thus exposing this capability to the Web as a native primitive solves real-world problems we have today.

Thanks,

-Anssi

[Mounir Lamouri]

On Tue, 4 Mar 2014, at 10:20, Peter Beverloo wrote:
> When I, as a developer, am faced with the decision of whether to build a
> native or a Web app, I'd start with listing the requirements of my app.
> Even when developing a Phone app, the ability to prevent accidental input
> isn't my top priority. I'd be more concerned with getting access to the
> person's contacts, keeping the device awake during the duration of the
> call, muting other audio playing on the device, maybe opening itself in
> the
> foreground if it isn't, and so on. Preventing accidental input could also
> be done by having focused UI, and requiring a more complicated gesture
> (e.g. sliding) to stop the conversation.

As much as I agree that proximity sensor is not a top priority, it is
definitely a nice to have. Hardware capabilities are a serious gap
between native and Web on mobile and having an API to access a new
hardware part is always a win. There are indeed a lot of features that
are missing to allow a developer to write a phone application on top of
the Web Platform. However, proximity sensor is part of those features
and I do not see why we should not consider it because some more complex
things would still be required.

It is indeed a fact that without keeping the device awake, your call
will unlikely be cut when you would be talking to your friend over a
WebRTC connection. But it would be as true for you hanging up on him/her
because your jaw pressed the hangup button.

> For air gestures, the granularity of the API often won't be high enough
> to
> effectively create gesture recognizers, unless the gestures are really
> basic -- slowly waving your hand three times, for example. The direction
> of
> a wave would not be detectable, either. Furthermore, it's noteworthy that
> all of these would require the device to be already unlocked with the
> page
> in the foreground, which makes most of the named use-cases unlikely.

You indeed need more than a proximity sensor to enable complex gestures.
A proximity sensor can't differentiate a hand going left from a hand
going right or a hand making a fist from an open hand. As far as I know
Samsung devices have a "gesture sensor" for that kind of stuff. This
said, even if the proximity sensor is pretty simple by itself, it still
allow applications to enable gestures. A quick wave in front of your
phone would pause the video you've been watching or the music, turn the
page of the book you were reading or simply lock the screen. Even if the
wave isn't needed per-se, it definitely improves the user experience.

> To give some more technical feedback which may be relevant...
>

> The specification defines two events, DeviceProximityEvent for indicating
> the distance to a nearby object, and UserProximityEvent for indicating
> the
> presence of a nearby object. This seems to be a security measure in case
> the user agent does not want to (or cannot) expose the precise distance

> between the device and a nearby object. The specification has a brief and

> very generic section on privacy considerations, but is inconclusive.
> Would
> we want to expose this to every website out there? Is there a need for a
> security mechanism?

Isn't the security mechanism pretty open here? We could imagine that the
UA would prompt after the content subscribed for the event. Another
solution would be to make the proximity estimation blurry. The events
could be sent based on the page visibility: they could be randomly
delayed for background tabs (to prevent tracking) or even not sent at
all.

> It would be a lot better to have a single event (UserProximityEvent?)
> with
> an additional property for a distance estimate when both known and it's
> safe to expose.

I don't know much about this API but from a quick glance I agree with
that.

> For Android, it's important to note that the platform does not make any
> guarantees about the values returned by the proximity sensor. Many
> sensors
> on Android seem to be based on the amount of lux received by a
> front-facing
> sensor, and estimate the distance based on the relative darkness. Others

> return fixed values depending on whether an object is far or near. The

> specification seems to be crafted around this, and defines that whatever
> values the platform happens to offer should just be forwarded to the Web
> application. How interoperable will these values be with, say, a mobile
> Apple or Windows device?

Given that the limitation is from the hardware and not the OS, with the
same hardware, you should see the same values regardless, right?

> If Blink were to support the Ambient Light API, a reasonable estimate of
> this functionality could be supported by a page monitoring the light
> conditions, and detecting when the amount of lux (steeply) decreases.
> When
> using WebRTC, a developer could approximate the distance of an object in
> front of the camera by measuring its velocity in relation to the device's
> orientation, but that can get complicated really quickly :-).

A bit complicated indeed :) Also quite a waste of CPU. It would be a
shame to have a sensor dedicated to detect user proximity and being
required to use another sensor (ambient light or camera) to emulate the
hardware capability.

On Tue, 4 Mar 2014, at 20:03, Kostiainen, Anssi wrote:
> > It would be a lot better to have a single event (UserProximityEvent?) with an additional property for a distance estimate when both known and it's safe to expose.
>
> Given an implementation of this feature is shipping in Firefox, I think
> we may not want to merge the interfaces at this time. If you feel
> strongly about this, I'd invite you to discuss this on the appropriate
> W3C mailing list (feel free to ping me off-the-list for more details).

I guess it would be more fair to point that the spec is now in CR
because Samsung implemented it in Webkit [1], thus it got to compatible
implementation I guess.
[1] https://bugs.webkit.org/show_bug.cgi?id=97630

-- Mounir

[Jake Archibald]

I think the only valid use-case here is 1. "phone on face" and "phone not on face". The API isn't expressive enough to cover the complex gestures in 2 or 3, except switching to ear speaker, which is again "phone on face". Also, the API isn't specific enough to cover the "phone on face" use-case, as the userproximity event isn't particular to that.

The use-cases in 2 sound better suited to devices like Leap Motion, Kinect, the S4's eye tracker, an API for non-touch interaction.

This particular API should be boiled down to navigator.screenCovered and naviagator.onscreencoveredchange, where screenCovered is true when "phone on face" (screenCovered is a terrible name, I'm sure there's better).

Then screenCovered can be specced more specifically as "Majority of screen covered, eg phoneonface".

As Peter says, to cover the RTC use-case we also need wake locks.

Jake.

On Mon, Mar 3, 2014 at 10:08 PM, Tim Volodine <timvo...@google.com> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.