Intent to implement and ship: Navigator.MediaDevices

[Henrik Grunell]

Contact emails

gru...@chromium.org

Spec

http://dev.w3.org/2011/webrtc/editor/archives/20140619/getusermedia.html#enumerating-devices

Summary

The spec adds mediaDevices attribute on Navigator. mediaDevices has

* An enumerateDevices function, which provides the same functionality as MediaStreamTrack.getSources, i.e. enumerates media devices. It adds audio output devices to the list.

* An ondevicechange event to notify when the set of devices available have changed.

(MediaStreamTrack.getSources is planned to be deprecated at a later point.)

Motivation

This has been added to the spec. The reason for adding audio output devices to the enumeration is to later allow developers to set the output device played out to. Exactly how to do that is under discussion. With output devices in the list, the enumerateDevices naming makes more sense, so does also moving it to Navigator.mediaDevices.

Compatibility Risk

Low.

The enumerate function itself is just a rename. Besides that adding audio output devices to the enumeration and the devicechange event. Audio output devices is not really useful until support for selecting sink device is implemented, so any changes to that should have a minor impact.

Ongoing technical constraints

None.

Will this feature be supported on all five Blink platforms (Windows, Mac, Linux, Chrome OS and Android)?

Yes.

OWP launch tracking bug?

None.

Implementation bug: https://code.google.com/p/chromium/issues/detail?id=388648

Link to entry on the feature dashboard

http://www.chromestatus.com/features/4906859072847872

Requesting approval to ship?

Yes.

/Henrik

[PhistucK]

​What is the Firefox stance? Was MediaStreamTrack.getSources implemented there as well?

Do you know of any plans of implementing this (navigator.mediaDevices) as well?

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Henrik Grunell]

MediaStreamTrack.getSources is not implemented in Firefox. I'm not aware of concrete plans of adding navigator.mediaDevices.

/Henrik

[Chris Palmer]

>>> http://dev.w3.org/2011/webrtc/editor/archives/20140619/getusermedia.html#enumerating-devices

Why is this necessary? Is there any use-case in which deciding which
media device to grant an origin access to should not be fully in
control of the user and the UA? Why should an origin need or want to
grovel around to discover my hardware configuration?

[Harald Alvestrand]

The user, via the UA, should be fully in control of what devices the application has access to.

This interface is mainly pointed to the "and then" - what happens after you've granted access.

Example: If you've told the application to use your USB web camera at a previous invocation, the grant has been stored, and the application wishes to default to the previuosly used camera if present, the application has a reasonable need to figure out if your USB web camera is connected or not.

Remember: For access to the most sensitive devices (camera and microphone), this API is just a rename of the getSources API, which we're already shipping.

[Chris Palmer]

On Tue, Jul 8, 2014 at 2:43 PM, Harald Alvestrand <h...@google.com> wrote:

> The user, via the UA, should be fully in control of what devices the
> application has access to.
> This interface is mainly pointed to the "and then" - what happens after
> you've granted access.
>
> Example: If you've told the application to use your USB web camera at a
> previous invocation, the grant has been stored, and the application wishes
> to default to the previuosly used camera if present, the application has a
> reasonable need to figure out if your USB web camera is connected or not.

I still don't see that the origin needs anything other than an opaque,
meaningful-only-to-the-browser token to support that use case. I'd
rather see:

1. origin requests a media device

2. to fulfill the request, the browser pops up a chooser UI

3. the user selects a device (possibly a null device)

4. the browser takes this selection and uses it to get a
handle/capability/descriptor to a device

5. the browser gives the handle to the origin, and (if the user wanted
the grant to be permanent) persists the grant by storing a GUID that
it can map back to the given device

> Remember: For access to the most sensitive devices (camera and microphone),
> this API is just a rename of the getSources API, which we're already
> shipping.

But if we're going to change anything, why not change it for the
better? I would have said the same thing about getSources if I had
been paying attention when that API was invented, too. :)

[Harald Alvestrand]

An opaque, meaningful-only-to-the-browser token is exactly what the origin gets before being granted permission. It's even origin-scoped, just to add another layer of isolation.

If the applicaiton wants to do something like doing an in-app device chooser with meaningful labels (which every single app seems to want), it has to get labels - but it can do that after being granted permission.

I'm all in favour of making things better.

I'm all AGAINST trying to change things by making private, Chrome-only discussions try to override compromises reached in open discussion fora like the W3C public-media-capture list.

[PhistucK]

You get the feedback of the Chrome team and then you should probably go to the standards body and propose this feedback (unless you disagree). These do not conflict, they complete each other.

The specification does not mention any permission that needs to be granted to enumerate the devices.

Have you implemented the flow so that the returned list is empty if the origin has no granted permission for getUserMedia, if not, will if throw? What happens?

Getting a list of devices (whether any detail, such as a label or anything else, is exposed or not) exposes the number of input and output devices the machine currently has and is subject to fingerprinting.

☆PhistucK

[Eric Seidel]

Phistuk: Harald is part of the (larger) Chrome team. :) The WebRTC
sub-team has not historically been as involved in these public
web-platform API discussions as we would like. Since Harald is coming
to us today, I would like to focus on constructive ways to help him be
more involved, not just scare them off.

Harald:

One name vs. another doesn't seem to matter terribly. It's unclear to
me from reading the spec and your email if this is the final name?
We've were recently burned by other specs changing API names (e.g.
DOMPromises) after we've shipped them (as it was done here to some
extent). It can be very hard to un-ship APIs.

Is it important to both implement and ship at this time? LGTM to
implement. But w/o any uses for this new API (my understanding is the
new bits like Audio Output aren't wired up yet?) it seems like we'd be
exposing ourselves to needless risk by shipping before we have
something to take advantage of this new functionality.


Re: permissions, etc. I agree with Harald (and others from past
threads) that the standards bodies are the place to argue spec
decisions. The Intent-To-Ship process is about deciding if Blink can
commit to an API long term.

Also, when we update to match the most recent spec, I presume we plan
to remove MediaStreamTrack.getSources? It looks like usage is quite
low:
http://www.chromestatus.com/metrics/feature/popularity#MediaStreamTrackGetSources

[PhistucK]

I know he is from the Chrome team, so I meant to say, "the rest of the Chrome team". I definitely did not mean or tried to scare him off - I was only explaining that these 'private' discussions are not meant to be authoritative or decisive regarding standards changing matters, but only as a probing first step and meant for getting even more feedback.

The use counter for getSources was only added five days ago -

https://codereview.chromium.org/370543002/

☆PhistucK

[Eric Seidel]

Thanks for the pointer! Looks like removal is already planned. Lets
hope it's successful.


Also, looks like the version we ship (getSources) is not even the one
in the stable spec:
http://www.w3.org/TR/mediacapture-streams/#widl-MediaStreamTrack-getSourceInfos-sequence-SourceInfo
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/modules/mediastream/MediaStreamTrack.idl&l=35

Hopefully the latest dev is really the final name:
http://dev.w3.org/2011/webrtc/editor/archives/20140619/getusermedia.html#enumerating-devices

Ouch.

[Harald Alvestrand]

The name got changed once because GetSources was not really a good name when it was extended to handle outputs too.

The name got changed again because we introduced the MediaDevices component (which was introduced in order to gain an event target, but had the added advantage that it minimized our intrusion on the rather bloated Navigator interface).

I really hope that we won't change it again, but each of those two changes seemed like a better choice at the time than living with the previous choice "forever" - I can make no promises.

Life when living with, not after, a standardization process.

[Henrik Grunell]

On Wed, Jul 9, 2014 at 8:36 AM, PhistucK <phis...@gmail.com> wrote:

You get the feedback of the Chrome team and then you should probably go to the standards body and propose this feedback (unless you disagree). These do not conflict, they complete each other.

The specification does not mention any permission that needs to be granted to enumerate the devices.

Have you implemented the flow so that the returned list is empty if the origin has no granted permission for getUserMedia, if not, will if throw? What happens?

This is expressed as:

"enumerateDevices

Collects information about the user agents available media input and output devices. The method must only return information that the script is authorized to access (TODO expand authorized)."

I believe there's a W3C bug somewhere with a proposed update for this todo, can't dig it up now. (Harald?)

It's implemented so that if no permission has been granted for getUserMedia, the label property is left empty, to reduce fingerprinting. This reflects the proposed update to the spec.

/Henrik

[PhistucK]

What Chris proposes (ask whether a specific device according to its formerly exposed GUID is available, instead of getting a list) sounds much better in terms of fingerprinting and privacy. What does everyone think about that?

☆PhistucK

[Eric Seidel]

API Owners met this afternoon and those in attendance agreed LGTM to implement, but not lgtm to ship at this time.

The major concern was that the api looks unstable, and likely to change again in the near future.  No reason to ship this flavor only to have to deprecate it again.

For example, the MediaDevices.enumerateDevices probably should return a promise based on modern patterns:

http://dev.w3.org/2011/webrtc/editor/archives/20140619/getusermedia.html#idl-def-MediaDevices

Seems like we can flesh out more of this via implementation and review before we commit to this API indefinitely.

[Harald Alvestrand]

Last time promises tried to push their way into the getusermedia API, they were met with somewhat fierce resistance.

There are a number of instances of the call / success-callback / failure-callback pattern in the getusermedia API; it does not make any sense to depart from this pattern for the enumerateDevices call only. Either we convert none, or we convert them all.

I've been wondering which is easiest/hardest: Polyfill promises on top of call/success/failure or polyfill call/success/failure on top of promises. This might affect the decision on which pattern ends up as part of the platform and which ends up as a commonly used polyfill.

 

[Adam Barth]

On Wed Jul 16 2014 at 12:21:22 AM, 'Harald Alvestrand' via blink-dev <blin...@chromium.org> wrote:

Last time promises tried to push their way into the getusermedia API, they were met with somewhat fierce resistance.

What was the reason for the resistance?  At the time folks were working on getusermedia, promises weren't ready.  Now they're on track to ship in every browser engine and are used extensively in new APIs.  There's even been a TAG finding to use promises in new APIs:

https://github.com/w3ctag/promises-guide

 

There are a number of instances of the call / success-callback / failure-callback pattern in the getusermedia API; it does not make any sense to depart from this pattern for the enumerateDevices call only. Either we convert none, or we convert them all.

I suspect you're going to end up converting them all.

I've been wondering which is easiest/hardest: Polyfill promises on top of call/success/failure or polyfill call/success/failure on top of promises. This might affect the decision on which pattern ends up as part of the platform and which ends up as a commonly used polyfill.

Both are equally possible.

Adam

[Chris Wilson]

+1.

I'd point out it's also possible to make a backwards-compatible API here that also moves to Promises:

    Promise getUserMedia (MediaStreamConstraints? constraints, NavigatorUserMediaSuccessCallback successCallback, NavigatorUserMediaErrorCallback errorCallback);

[PhistucK]

The real place of getUserMedia is under navigator.mediaDevices. It is kept under navigator.getUserMedia (as well? instead?) for legacy and backward compatibility purposes.

Perhaps the final, non prefixed API should move to navigator.mediaDevices.getUserMedia(constraints) and return a promise?

adapter.js (is it widely used? does anyone know?) can bridge the differences anyway and we would be left with a clean, modern and relatively encapsulated (under navigator.mediaDevices) API.

☆PhistucK

[Kenneth Auchenberg]

Great to see the API being moved under navigator, and become more streamlined.

I have a question regarding devices and labels. The current behavior in Chrome is that device labels isn't returned for non-https origins (ex. http://jsbin.com/guhomite/3/edit).

This behavior isn't specified in the spec (to my knowledge), and can be quite frustrating, as no errors are thrown and other properties are returned, which results in a half-broken UI as the device-list label-less, and therefore unusable for the end-user.

What is the reasoning for this behavior?

/k

-
Kenneth Auchenberg
+45 53 22 22 33

[Harald Alvestrand]

Labels leak information that is useful in fingerprinting.

After batting the question of "how much fingerprinting is enough" back and forth a few times, the group ended up with exposing labels to the drive-by web only after a permission prompt had been processed by the user.

HTTP vs HTTPS has nothing to do with it - but it's more likely that you have a stored permission with HTTPS, since with HTTP, you don't get the offer to store permissions.

[Chris Palmer]

On Tue, Aug 5, 2014 at 4:27 AM, Harald Alvestrand <h...@google.com> wrote:

> HTTP vs HTTPS has nothing to do with it - but it's more likely that you have
> a stored permission with HTTPS, since with HTTP, you don't get the offer to
> store permissions.

Exactly.

That said, I can imagine opening up the "remember my permission prompt
responses" functionality to all securely-transported origins, not just
HTTPS. FWIW.

[Alex Russell]

Please do.

Also, what's the relationship between the output media device and the implicit output device from the WebAudo spec?

[Ian Hickson]

On Tue, 5 Aug 2014, 'Harald Alvestrand' via blink-dev wrote:
>
> After batting the question of "how much fingerprinting is enough" back
> and forth a few times, the group ended up with exposing labels to the
> drive-by web only after a permission prompt had been processed by the
> user.

Does anyone know what the proposed prompt should be? Has it been
user-tested? I'm struggling to come up with wording that would explain the
problem in a user-understandable way.

--
Ian Hickson U+1047E )\._.,--....,'``. fL
http://ln.hixie.ch/ U+263A /, _.. \ _\ ;`._ ,.
Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'

[PhistucK]

I believe Harald is referring to the getUserMedia permission and not to a specific permission for querying available user media devices.

☆PhistucK

[Ian Hickson]

On Mon, 18 Aug 2014, PhistucK wrote:
> On Mon, Aug 18, 2014 at 9:06 PM, Ian Hickson <i...@hixie.ch> wrote:
> > On Tue, 5 Aug 2014, 'Harald Alvestrand' via blink-dev wrote:
> > >
> > > After batting the question of "how much fingerprinting is enough"
> > > back and forth a few times, the group ended up with exposing labels
> > > to the drive-by web only after a permission prompt had been
> > > processed by the user.
> >
> > Does anyone know what the proposed prompt should be? Has it been
> > user-tested? I'm struggling to come up with wording that would explain
> > the problem in a user-understandable way.
>

> I believe Harald is referring to the getUserMedia permission and not to
> a specific permission for querying available user media devices.

I'm confused.

Say I go to a site, and the site wants to get video from my USB cam, or
play audio to my remote speakers. What's the expected interface?

[PhistucK]

You can get a label-less list (pretty useless) of devices using navigator.mediaDevices.enumerateDevices, you call getUserMedia, you get a permission request, always accept and the labels are exposed from now on using navigator.mediaDevices.enumerateDevices.

Correct me if I am wrong, Harald.

☆PhistucK

[Chris Palmer]

On Mon, Aug 18, 2014 at 1:52 PM, Ian Hickson <i...@hixie.ch> wrote:

> Say I go to a site, and the site wants to get video from my USB cam, or
> play audio to my remote speakers. What's the expected interface?

I would like to see something like this:

1. Web content in (https, example.com, 443) creates a div, button, or
other element whose onclick handler calls getUserMedia. (Or whatever.)

2. The browser handles that call by opening up a chooser, in
trustworthy native UI (not web contents) giving the user a choice of
which input device (or pseudo-device, including still image/avatar,
null device, whatever) to expose to the origin. There is a checkbox to
the effect of: "[x] Remember this decision for [ Good HTTPS Icon ]
example.com".

3. (https, example.com, 443) gets an opaque handle to the device.

On could argue, and in Chrome we generally try to, that broken HTTPS,
HTTP, or otherwise non-secure origins shouldn't get persistent access,
i.e. no [x] Remember checkbox. (Or even that non-secure origins don't
even get to ask, or only get access to pseudo-devices, or, or, ...)

[Ian Hickson]

On Mon, 18 Aug 2014, PhistucK wrote:
>
> You can get a label-less list (pretty useless) of devices using
> navigator.mediaDevices.enumerateDevices

Note that that gives a few bits of fingerprintability on its own (the
number of devices, at least).

> you call getUserMedia, you get a permission request

What's the label of that permission request? That's my question. I can't
figure out how to phrase it in a useful informative way that won't be
ignored or considered ridiculous.

> always accept and the labels are exposed from now on using
> navigator.mediaDevices.enumerateDevices. Correct me if I am wrong,
> Harald.

I think the UI flow suggested by Chris makes way more sense.

[PhistucK]

Of course, ​I agree about the fingerprint-ability, I was merely stating the current (or currently intended) situation.​

You can go to https://apprtc.appspot.com to see the permission request right now. :)

☆PhistucK

[Harald Alvestrand]

I think the flow Chris is describing *is* the current flow, to a close approximation.

On Mon, Aug 18, 2014 at 11:34 PM, Chris Palmer <pal...@google.com> wrote:

On Mon, Aug 18, 2014 at 1:52 PM, Ian Hickson <i...@hixie.ch> wrote:

> Say I go to a site, and the site wants to get video from my USB cam, or
> play audio to my remote speakers. What's the expected interface?

I would like to see something like this:

1. Web content in (https, example.com, 443) creates a div, button, or
other element whose onclick handler calls getUserMedia. (Or whatever.)

This is the existing flow.

 

2. The browser handles that call by opening up a chooser, in
trustworthy native UI (not web contents) giving the user a choice of
which input device (or pseudo-device, including still image/avatar,
null device, whatever) to expose to the origin. There is a checkbox to
the effect of: "[x] Remember this decision for [ Good HTTPS Icon ]
example.com".

This is the existing flow, with the prompt being "<website> would like to use your camera and microphone".

We had a chooser in an earlier version; UI people were not happy, we abandoned it. (I think Firefox still has it.)

We also had a "remember this choice" on an earlier version; UI people were not happy; we abandoned it, and will now always remember for HTTPS and never remember for HTTP.

 

3. (https, example.com, 443) gets an opaque handle to the device.

This is the current flow; the opaque handle is called a MediaStream.

 

On could argue, and in Chrome we generally try to, that broken HTTPS,
HTTP, or otherwise non-secure origins shouldn't get persistent access,
i.e. no [x] Remember checkbox. (Or even that non-secure origins don't
even get to ask, or only get access to pseudo-devices, or, or, ...)

So the difference between current Chrome and what Chris is suggesting is the amount of information and user choices presented in the permission-requesting prompt.

[Chris Palmer]

On Aug 19, 2014 3:50 AM, "Harald Alvestrand" <h...@google.com> wrote:

> We had a chooser in an earlier version; UI people were not happy, we abandoned it. (I think Firefox still has it.)
> We also had a "remember this choice" on an earlier version; UI people were not happy; we abandoned it, and will now always remember for HTTPS and never remember for HTTP.

Can you be more specific about what people didn't like about it?

> So the difference between current Chrome and what Chris is suggesting is the amount of information and user choices presented in the permission-requesting prompt.

That's a pretty big difference. :)

I feel strongly that the current permissions system is a security UX anti-pattern and that we can serve users better.

[Ian Hickson]

On Tue, 19 Aug 2014, PhistucK wrote:
>
> You can go to https://apprtc.appspot.com to see the permission request
> right now. :)

That site didn't work at all. It picked completely the wrong camera, and
there was no obvious way for me to tell it the right one. Also, what about
a site that requires a camera, but for which I just want to pass a looping
stock video rather than giving the site access to my actual camera?

[Chris Palmer]

On Aug 19, 2014 10:00 AM, "Ian Hickson"

> That site didn't work at all. It picked completely the wrong camera, and
> there was no obvious way for me to tell it the right one. Also, what about
> a site that requires a camera, but for which I just want to pass a looping
> stock video rather than giving the site access to my actual camera?

Indeeeeeed.

[PhistucK]

I am not sure WebRTC (in Chrome, anyway) allows video that does not come from a capture device (yet?).

☆PhistucK

[Elliott Sprehn]

I'm not aware of any native applications that allow that, I don't think we need to add that kind of thing (or optimize for it) on the web.

[Jeffrey Yasskin]

I think the structure of that argument isn't right. :)

One thing you can do on the web that you can't do with native
applications is use things you don't trust or don't fully trust.
Allowing non-captured video on the web would double down on that
advantage.

I can't say how high a priority this should be in the case of
audio/video. Probably lower priority than letting users choose *which*
of several cameras/microphones to grant access to.

[Chris Palmer]

On Tue, Aug 19, 2014 at 12:00 PM, Jeffrey Yasskin <jyas...@chromium.org> wrote:

> One thing you can do on the web that you can't do with native
> applications is use things you don't trust or don't fully trust.
> Allowing non-captured video on the web would double down on that
> advantage.

Agreed.

> I can't say how high a priority this should be in the case of
> audio/video. Probably lower priority than letting users choose *which*
> of several cameras/microphones to grant access to.

Probably; but also, we can enjoy not having to prioritize. :) The same
mechanism that allows users to choose their preferred camera/mic also
allows them to choose their preferred static image or pre-recorded
loop or solid color or whatever. If the UX of the my OS' desktop
background selector can do it, ...

[fe...@google.com]

On Tuesday, August 19, 2014 3:50:59 AM UTC-7, Harald Alvestrand wrote:

I think the flow Chris is describing *is* the current flow, to a close approximation.

On Mon, Aug 18, 2014 at 11:34 PM, Chris Palmer <pal...@google.com> wrote:

On Mon, Aug 18, 2014 at 1:52 PM, Ian Hickson <i...@hixie.ch> wrote:

> Say I go to a site, and the site wants to get video from my USB cam, or
> play audio to my remote speakers. What's the expected interface?

I would like to see something like this:

1. Web content in (https, example.com, 443) creates a div, button, or
other element whose onclick handler calls getUserMedia. (Or whatever.)

This is the existing flow.

 

2. The browser handles that call by opening up a chooser, in
trustworthy native UI (not web contents) giving the user a choice of
which input device (or pseudo-device, including still image/avatar,
null device, whatever) to expose to the origin. There is a checkbox to
the effect of: "[x] Remember this decision for [ Good HTTPS Icon ]
example.com".

This is the existing flow, with the prompt being "<website> would like to use your camera and microphone".

We had a chooser in an earlier version; UI people were not happy, we abandoned it. (I think Firefox still has it.)

We also had a "remember this choice" on an earlier version; UI people were not happy; we abandoned it, and will now always remember for HTTPS and never remember for HTTP.

FWIW, I believe that chooser will be coming back. We have mocks and general support for it.

[fe...@google.com]

I don't understand. If the flows are the same, why do we also need to provide the ability to enumerate devices (which Chris's proposal omits)?

On Tuesday, August 19, 2014 3:50:59 AM UTC-7, Harald Alvestrand wrote: