Intent to remove: sensor APIs

[Anne van Kesteren]

As a follow-up to the Ambient Light Sensor API thread, which ended up
not really concluding, I hereby suggest we remove the various sensor
APIs from our code base. Flipping the preference first to make sure
there's no undue impact on web content and quick reversal is possible
and then removing the code in a later release.

This affects these APIs:

* Ambient light
* Proximity
* Device orientation

The rationale:

* These APIs have various privacy leaks, including violating the
same-origin policy, without the user being informed.
* These APIs do not match the current standards for sensor APIs and
some are incompatible with what is being shipped by Chrome (e.g.,
device orientation).
* There's no interest to address these shortcomings. (Mostly in the
sense of engineering resources and having other problems to tackle
first.)
* As these are event-driven APIs the compatibility impact should be
minimal to none. The events simply won't fire.

The bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1359076


--
https://annevankesteren.nl/

[Ben Kelly]

On Mon, Jul 24, 2017 at 5:10 AM, Anne van Kesteren <ann...@annevk.nl> wrote:

> * Device orientation
>

Isn't this one required to build a decent web experience on mobile for some
sites? It seems pretty common on mobile to adjust the UX based on whether
the device is in portrait/landscape orientation.

[Blair MacIntyre]

I was just about to say the same thing. This API is essential for our AR work; the fact that Firefox is different than other browsers is problematic, but there are javascript libraries that help with it. Getting rid of it would be really bad.

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Mike Hoye]

I have a sense that as AR gets richer and more nuanced that ambient
light and proximity sensing will become important as well, even if we're
not there yet.

- mhoye

[Blair MacIntyre]

True, true. For example, if the ambient light sensing could deliver the kind of “estimation of ambient lighting” that Apple’s ARKit does, we could use that in rendering.

But, one question will be: what of these capabilities should just be part of “WebAR”, and which can be used effectively independent of WebAR? Especially when we think of issues discussed on this thread (e.g., threat models and security) having “things needed for AR” folded into WebAR, and accessible in the context of whatever permission model WebAR ends up using may be the way we want to go.

--
Blair MacIntyre
Principal Research Scientist
bmaci...@mozilla.com <mailto:bmaci...@mozilla.com>

> On Jul 24, 2017, at 11:07 AM, Mike Hoye <mh...@mozilla.com <mailto:mh...@mozilla.com>> wrote:
>
>
> I have a sense that as AR gets richer and more nuanced that ambient light and proximity sensing will become important as well, even if we're not there yet.
>
> - mhoye
>
> On 2017-07-24 10:39 AM, Blair MacIntyre wrote:
>> I was just about to say the same thing. This API is essential for our AR work; the fact that Firefox is different than other browsers is problematic, but there are javascript libraries that help with it. Getting rid of it would be really bad.
>>

>>> On Jul 24, 2017, at 9:57 AM, Ben Kelly <bke...@mozilla.com <mailto:bke...@mozilla.com>> wrote:

>>>
>>> On Mon, Jul 24, 2017 at 5:10 AM, Anne van Kesteren <ann...@annevk.nl <mailto:ann...@annevk.nl>> wrote:
>>>
>>>> * Device orientation
>>>>
>>> Isn't this one required to build a decent web experience on mobile for some
>>> sites? It seems pretty common on mobile to adjust the UX based on whether
>>> the device is in portrait/landscape orientation.
>>> _______________________________________________
>>> dev-platform mailing list

>>> dev-pl...@lists.mozilla.org <mailto:dev-pl...@lists.mozilla.org>

>>> https://lists.mozilla.org/listinfo/dev-platform
>> _______________________________________________
>> dev-platform mailing list

>> dev-pl...@lists.mozilla.org <mailto:dev-pl...@lists.mozilla.org>
>> https://lists.mozilla.org/listinfo/dev-platform
>
>

[Anne van Kesteren]

Please consider the request to remove device orientation retracted for
now. We'll still need to figure out some kind of long term plan for
that API though. WebVR building on it through libraries that abstract
away the browser incompatibilities will just make it harder to fix the
underpinnings going forward. (And there's always the risk that folks
don't use libraries and code directly against what Chrome ships. Seems
likely even.)

On Mon, Jul 24, 2017 at 5:07 PM, Mike Hoye <mh...@mozilla.com> wrote:
> I have a sense that as AR gets richer and more nuanced that ambient light
> and proximity sensing will become important as well, even if we're not there
> yet.

I'm not sure that's a good reason to keep the current APIs around
though. Ambient light in particular leaks cross-origin data. It's
rather surprising we haven't acted on that thus far. And none of what
we ship ended up being standardized as such. If we actually think we
need these APIs we should put in the effort to solve the security and
standardization issues.


--
https://annevankesteren.nl/

[Enrico Weigelt, metux IT consult]

On 24.07.2017 13:57, Ben Kelly wrote:
> On Mon, Jul 24, 2017 at 5:10 AM, Anne van Kesteren <ann...@annevk.nl>

wrote:
>
>> * Device orientation
>>
>
> Isn't this one required to build a decent web experience on mobile
for some
> sites?

Could you please define "decent web experience" ?

Maybe I'm just old, but I absolutely don't want an website know anything
about my local devices. Window size and resolution is fine, but ambient
conditions seriously crossed the red line of what I'd ever allow on my
machines.

I've already started patching out that stuff.

Same for things like USB devices, or all that "cloud" stuff.


--mtx

[Enrico Weigelt, metux IT consult]

On 24.07.2017 15:07, Mike Hoye wrote:
>
> I have a sense that as AR gets richer and more nuanced that ambient

Are we still talking about browsers ?


--mtx

[Kearwood Kip Gilbert]

Please note that disabling the Device Orientation API will also effectively disable the “WebVR Polyfill”. The “WebVR Polyfill” is a javascript library that allows browser that otherwise don’t support VR (ie, Fennec) to use “Cardboard” VR holders to create simple VR experiences. Usage of these VR holders with non-VR capable browsers is quite prolific, with these holders found in most department stores.

The WebVR API when implemented directly by the browser (ie. Firefox Desktop) negates the need of the Device Orientation API, as it exposes the tracking data from the VR positional sensors in a more direct fashion.

A non-VR use case that affects 90% of users is the “magic window” effects. These are most commonly used by sites such as Facebook to give some degree of freedom in viewing a 360 panorama video or photo by moving the phone around.

Combined with media capture API’s the orientation API is also essential for implementing things like “Google Goggle” or Yelp’s Monocle on the web platform.

Cheers,
- Kip

_______________________________________________
dev-platform mailing list
dev-pl...@lists.mozilla.org

https://lists.mozilla.org/listinfo/dev-platform

[Blair MacIntyre]

Yes.

There are plenty of websites that use deviceorientation, for example to let you pan around a panographic photo or a 3D “scene” using the orientation of your mobile device.

Most websites that use WebVR prefer to use “real” WebVR APIs, but on most mobile browsers (which don’t support them) use deviceorientation to support a “3DOF” (just orientation) view in the world.

We are working on adding AR capabilities to the browser, and this will (similarly) need to know device orientation.

[Enrico Weigelt, metux IT consult]

On 24.07.2017 20:43, Kearwood Kip Gilbert wrote:
> Please note that disabling the Device Orientation API will also
> effectively disable the “WebVR Polyfill”. The “WebVR Polyfill” is a
> javascript library that allows browser that otherwise don’t support VR
> (ie, Fennec) to use “Cardboard” VR holders to create simple VR
> experiences.

Am I the only one here still remembering that web-browsers used
to be HTTP clients w/ hypertext display ?

The whole VR stuff might be nice for some niches (eg. CAD stuff, etc),
but I wonder what that got to do w/ hypertext ...

> A non-VR use case that affects 90% of users is the “magic window”
> effects. These are most commonly used by sites such as Facebook to give
> some degree of freedom in viewing a 360 panorama video or photo by
> moving the phone around.

A website (especially FB) getting device positioning information
delivered by the browser - that is *really* scary!

The wireless tracking (even w/o gps) already is a massive problem,
we need to care about (not a browser topic), the situation is already
worse than in the previous socialist regimes. We shouldn't make that
even worse.

> Combined with media capture API’s the orientation API is also essential
> for implementing things like “Google Goggle” or Yelp’s Monocle on the
> web platform.

Sorry, but these kind misfeatures are a total security nightmare,
way beyond the red line. Bad enought the people collaborate w/ mass
sourveillance by using games like pokemon go (which, IMHO should be
prohibited). But infecting browers with that is magnitudes worse.


--mtx

[Enrico Weigelt, metux IT consult]

On 24.07.2017 20:46, Blair MacIntyre wrote:

> We are working on adding AR capabilities to the browser, and this will (similarly)
> need to know device orientation.

Please make sure, we can easily compile completely w/o that.


--mtx

[Blair MacIntyre]

I’m not sure what you’re asking: I’ve been using the deviceorientation API like this for many years, as have plenty of other people. It’s absolutely needed.

--
Blair MacIntyre
Principal Research Scientist
bmaci...@mozilla.com <mailto:bmaci...@mozilla.com>

[Anne van Kesteren]

On Mon, Jul 24, 2017 at 6:11 PM, Anne van Kesteren <ann...@annevk.nl> wrote:
> Please consider the request to remove device orientation retracted for
> now. We'll still need to figure out some kind of long term plan for
> that API though. WebVR building on it through libraries that abstract
> away the browser incompatibilities will just make it harder to fix the
> underpinnings going forward. (And there's always the risk that folks
> don't use libraries and code directly against what Chrome ships. Seems
> likely even.)

Small update: we'll start by just disabling proximity. Disabling
ambient light will follow soon after, but is a little trickier as we
use the web-facing API in the Firefox for Android frontend.
(Suggestions for fixing the orientation interoperability mess are
still welcome!)


--
https://annevankesteren.nl/

[Salvador de la Puente]

I strongly encourage you to take a look at the telemetry stats regarding
the usage of deviceorientation API and other. I don't know the penetration
of proximity and ambient light APIs but deviceorientation is definitively
used.

Please, consider twice before taking a final decision.

[Frederik Braun]

As mentioned in thread, we will not disable deviceorientation.
Please see below.

[Enrico Weigelt, metux IT consult]

On 02.08.2017 13:01, Salvador de la Puente wrote:
> I strongly encourage you to take a look at the telemetry stats regarding
> the usage of deviceorientation API and other. I don't know the penetration
> of proximity and ambient light APIs but deviceorientation is definitively
> used.

Just curious: for what exactly is it used ?

For rendering / GUI, I'd assume that the job of the windowing system /
compositor - the application would just see a window geometry change.

Making that information visible to websites (even worse: movement
tracking via g-sensor, etc), definitively looks like security nightmare
which even the Stasi never dared dreaming of.

--mtx

[Michael Hoye]

On Aug 2, 2017 15:54, "Enrico Weigelt, metux IT consult" <

enrico....@gr13.net> wrote:



Making that information visible to websites (even worse: movement
tracking via g-sensor, etc), definitively looks like security nightmare
which even the Stasi never dared dreaming of.

You need to dial this rhetoric back about 100%. It is not acceptable to
bring even an implied accusation like that to a technical discussion, or
indeed any conversation at all, at Mozilla.

We're always happy to listen to honest criticism and walk back our
mistakes, but we are going have those discussions without demeaning the
work or comparing the people doing that work to volkscryptopolitzei
collaborators.

I encourage you to read our community participation guidelines carefully,
and to take them to heart before continuing.

Thank you.

-mhoye

[Blair MacIntyre]

Are we still talking about deviceorientation?

It’s used to determine the 3D orientation of the device, so that we can tell the direction it is facing. Developers use it to render 3D graphics (WebGL or CSS3D using perspective DIV) around the user. e.g., look at one of my project samples, like https://samples.argonjs.io/directionsWebGL <https://samples.argonjs.io/directionsWebGL>, which uses device location and deviceorientation (this simple samples puts 3D labels in the cardinal directions, and uses the position to illuminate them based on the current sun location). The WebVR polyfill uses it to determine viewing direction, to simulate 3D device orientation.

It’s used for panoramic image viewing (orient the pano with the camera movement), and google street view uses it for similar motion control.

Regarding security: perhaps it is, I have seen discussions of this sort. But, it would seem that ship sailed when the W3C approved it, and now it’s common and assumed and relied upon. Removing it in Firefox would render Firefox incompatible with a growing use of the web, especially mobile (including Windows tablets). This might be a discussion the security team wants to have, I guess: is the Firefox team worried enough about the threats opened by this API to justify breaking a large class of applications, and making Firefox unusable for AR/VR moving forward.

> On Aug 2, 2017, at 9:54 AM, Enrico Weigelt, metux IT consult <enrico....@gr13.net> wrote:
>
> On 02.08.2017 13:01, Salvador de la Puente wrote:
>> I strongly encourage you to take a look at the telemetry stats regarding
>> the usage of deviceorientation API and other. I don't know the penetration
>> of proximity and ambient light APIs but deviceorientation is definitively
>> used.
>
> Just curious: for what exactly is it used ?
>
> For rendering / GUI, I'd assume that the job of the windowing system /
> compositor - the application would just see a window geometry change.
>

> Making that information visible to websites (even worse: movement
> tracking via g-sensor, etc), definitively looks like security nightmare
> which even the Stasi never dared dreaming of.
>

> --mtx
>

[Anne van Kesteren]

On Wed, Aug 2, 2017 at 4:39 PM, Blair MacIntyre <bmaci...@mozilla.com> wrote:
> Are we still talking about deviceorientation?

As I said twice and Frederik repeated, we're not, other than asking if
anyone has a plan for how to make it interoperable. Note that it's far
from a W3C standard: https://www.w3.org/TR/orientation-event/. Doesn't
seem like anything got approved there.


--
https://annevankesteren.nl/

[Blair MacIntyre]

> On Wed, Aug 2, 2017 at 4:39 PM, Blair MacIntyre <bmaci...@mozilla.com> wrote:
>> Are we still talking about deviceorientation?
>
> As I said twice and Frederik repeated, we're not, other than asking if
> anyone has a plan for how to make it interoperable.

Yes, I know; I was just responding to Enrico’s question. :)

> Note that it's far
> from a W3C standard: https://www.w3.org/TR/orientation-event/. Doesn't
> seem like anything got approved there.

Fair enough, sorry for my incorrect assertion, I should have checked! So, this is an example of the danger, perhaps, of everybody implementing a working proposal before it’s approved, and then having it adopted and used widely.

[Enrico Weigelt, metux IT consult]

On 02.08.2017 14:39, Blair MacIntyre wrote:

> It’s used for panoramic image viewing (orient the pano with the camera
> movement), and google street view uses it for similar motion control.

Okay, why not adding a generic interface for controlling the virtual
view direction ? So, the user/operator could decide how to control it
(keys, mouse gestures, external 3d positioning devices, etc).

Anyways, I wonder how the current approach would work at all with
non-portable devices. And the idea of having to physically turn around
just to see different perspectives seems really weird to me.

> Regarding security: perhaps it is, I have seen discussions of this
> sort.

Allowing webites to track individual movements, IMHO qualifies more
than just an "perhaps". Do you feel well with the idea of being tracked
with every single footstep ?

> But, it would seem that ship sailed when the W3C approved it, and
> now it’s common and assumed and relied upon. Removing it in Firefox
> would render Firefox incompatible with a growing use of the web,

Okay, we now know that was a wrong assumption - but even it was
really was approved: does that mean we have to support anything that
some beaurocrats find a good idea ?

In the past, there have been lots of standards that peopel stopped
supporting / complying to, because they turned out to be a bad idea.
I still remember when leaded fuel was standard - meanwhile it's even
prohibited (except for a few old timers). I also remember when IPX
or DecNet have been de-facto standards. Flash has been a de-facto
standard, it was a security nightmare, and it took a long time to get
rid of it.

What would you do if W3C decided that web applications shall be allowed
to execute arbitrary binaries with user's full privileges ?
(actually, I wouldn't be surprised if Goverments making such laws ...)

> especially mobile (including Windows tablets).

What would be the actual impact ? A few websites that rely on that
won't work completely anymore. Actually, haven't seen single one yet.

Is it okay, to sacrify the security of all users just for the joy of
that small minority ?

> This might be a discussion the security team wants to have, I guess:
> is the Firefox team worried enough about the threats opened by this
> API to justify breaking a large class of applications, and making
> Firefoxunusable for AR/VR moving forward.

Suggestive question. And implies that this "large class of applications"
actually exists already.

I'd prefer asking: is it okay to sacrifice security for some niche
features ?

OTOH, I'd also question whether that AR/VR really belongs into a
browser, and what's the costs and risks of that. (some countries
already started legislation against AR, most notably Pokemon Go, for
good reasons ...).

Certainly, AR/VR has it's use in some industrial applications or
entertainment machines, but I doubt Browsers and HTML are particular
well suited for those jobs.



--mtx