gUM combined with media device enumeration/selection: problematic to make consistent UI

[jeremy...@gmail.com]

Let's say I want to build a WebRTC app and allow users to enumerate and select media devices in the page. I want this UI as consistent as possible across browsers.

Right now, the situation is not good in Firefox. After an initial gUM call (so I have labels for my enumerated devices), the user flow looks like this:

1. Webpage has UI for enumerating, selecting devices
2. User changes mic or cam
3. User did not select "always allow", so gUM dialog shows again, with its own selection options
4. User must click share to share a second time.

This is klunky and weird for users--they made a selection, so the secondary gUM dialog showing up is really confusing. If the user did select "always allow" then this custom selection works how you'd expect: their changes are reflected immediately.

You'd think a workaround would almost be for the web page to simply have a button to change mic/speakers, so the gUM dialog shows and user can select them there. However, that breaks down if the user DID select "always allow" because then gUM shows no dialog and there's no way to change devices at all.

Chrome doesn't suffer from this problem, because after an initial gUM request on the page, it allows unprompted access to that page--basically, "always allow" being the default.

My question is whether or not there's any work scheduled in Firefox to resolve this ambiguity? Even if they didn't select "always allow," does it really need to prompt with every change? Could it just prompt once when the page is loaded instead? It is great to be able to enumerate mic/camera and select the ones you want, but that feature is extremely hard (bordering on impossible to communicate clearly to users) to implement in a sane way with how gUM currently prompts in Firefox.

You can test all of this at https://webrtc.github.io/samples/src/content/devices/input-output/

[Maire Reavy]

The best person to reply to this is Jan-Ivar (jib on IRC); he's on PTO
until after New Years. I'll point this thread out to him when he's back.

Thanks.
-Maire

> _______________________________________________
> dev-media mailing list
> dev-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-media
>



--
Maire Reavy
mre...@mozilla.com

[jno...@hirevue.com]

After consulting the w3g github project (https://github.com/w3c/mediacapture-main/issues/299#issuecomment-168856020), I see two potential solutions:

1. Set the label field on the media stream track to the MediaDeviceInfo's label
2. Implement MediaStreamTrack.getSettings(), which would include a deviceId that maps directly to a MediaDeviceInfo deviceId

Second is probably a better solution. First is probably faster.

[Jan-Ivar Bruaroey]

On 12/17/15 1:13 PM, jeremy...@gmail.com wrote:
> This is klunky and weird for users--they made a selection, so the
> secondary gUM dialog showing up is really confusing.

> You can test all of this at
> https://webrtc.github.io/samples/src/content/devices/input-output/

Look closely at the Firefox permission dialog here, and you'll see it
doesn't actually let users change the camera or microphone (because the
site uses `exact` constraints). Click on the cam/mic, and the only other
option is "No Camera/Microphone" (a bit of a dud choice at the moment),
so the users are merely being informed about which camera/mic is being
sought here.

I agree it is confusing, which could be remedied with better UX in the
desktop versions of Firefox. If you wouldn't mind, please file a bug on
the confusing UX here
https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&component=Device%20Permissions

For example, Firefox for Android doesn't have this problem because it
omits the selector entirely in this case. On the other hand, it omits
telling users whether they're sharing the front or back camera. Showing
cam/mic in plain text when they can't be changed might be better.

> Even if they didn't select "always allow," does it really need to
> prompt with every change?

The primary purpose of the prompt is to request user permission. Firefox
is being protective of users here by not forcing "always allow" on
users. The selector is a secondary feature.

Even once we fix the desktop UX, arguments have been made that the
overall experience is laborious when sites build in device selection,
but no great solution has been found so far.
See https://bugzil.la/1212996 for more.

..: Jan-Ivar :.

[Jan-Ivar Bruaroey]

We're aiming to get to these this cycle.

..: Jan-Ivar :.

[jno...@hirevue.com]

On Thursday, January 7, 2016 at 9:22:15 AM UTC-7, Jan-Ivar Bruaroey wrote:
> Look closely at the Firefox permission dialog here, and you'll see it
> doesn't actually let users change the camera or microphone (because the
> site uses `exact` constraints).

Interesting--is that "exact" constraint documented somewhere? Does Chrome support it as well?

This would roughly solve my problem, because it'd be possible to know as a developer if they opted not to share something because the resulting MediaStreamTrack would contain no corresponding track.

> > Even if they didn't select "always allow," does it really need to
> > prompt with every change?
>
> The primary purpose of the prompt is to request user permission. Firefox
> is being protective of users here by not forcing "always allow" on
> users. The selector is a secondary feature.

I am not advocating "always allow". My point: is it really necessary for someone to "allow" repeatedly on the same page load?

Example: I browse to www.myVideoConferencingSite.com, which requests access to mic and camera. The user grants it. Later, the user *changes* mic and camera. This causes another request to access mic and camera, which is tedious since the user already granted permission to mic/camera.

I see the exact subject being discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1212996 , but the reality is Chrome already functions the way people are arguing amounts to "collateral grant." I disagree with this because it's a questionable assertion that users see a "privacy" difference from one mic/camera to another.

Also, there are very real implications for sites using WebRTC, because users effectively aren't streaming media while the change is occurring. On Chrome it's a quick blip, but on Firefox it can be seconds or more. Hopefully this situation will be improved with the ORTC spec and the ability to change camera/mic without affecting streaming, but for Firefox, the user is literally gone from a live stream for as long as it takes them to navigate that extra UI.

I'll post in that defect. Thanks for the response.

[Martin Thomson]

Think of this not as "can I use your camera", but as "pick a camera
that I can use". When you pick a file for upload, you don't give the
site permission to the subsequently upload all the other files in that
directory. The "remember my choice" option is what is special here.

On Fri, Jan 8, 2016 at 4:38 AM, <jno...@hirevue.com> wrote:
> On Thursday, January 7, 2016 at 9:22:15 AM UTC-7, Jan-Ivar Bruaroey wrote:

>> Look closely at the Firefox permission dialog here, and you'll see it
>> doesn't actually let users change the camera or microphone (because the
>> site uses `exact` constraints).
>

> Interesting--is that "exact" constraint documented somewhere? Does Chrome support it as well?
>
> This would roughly solve my problem, because it'd be possible to know as a developer if they opted not to share something because the resulting MediaStreamTrack would contain no corresponding track.
>

>> > Even if they didn't select "always allow," does it really need to
>> > prompt with every change?
>>
>> The primary purpose of the prompt is to request user permission. Firefox
>> is being protective of users here by not forcing "always allow" on
>> users. The selector is a secondary feature.
>

> I am not advocating "always allow". My point: is it really necessary for someone to "allow" repeatedly on the same page load?
>
> Example: I browse to www.myVideoConferencingSite.com, which requests access to mic and camera. The user grants it. Later, the user *changes* mic and camera. This causes another request to access mic and camera, which is tedious since the user already granted permission to mic/camera.
>
> I see the exact subject being discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1212996 , but the reality is Chrome already functions the way people are arguing amounts to "collateral grant." I disagree with this because it's a questionable assertion that users see a "privacy" difference from one mic/camera to another.
>
> Also, there are very real implications for sites using WebRTC, because users effectively aren't streaming media while the change is occurring. On Chrome it's a quick blip, but on Firefox it can be seconds or more. Hopefully this situation will be improved with the ORTC spec and the ability to change camera/mic without affecting streaming, but for Firefox, the user is literally gone from a live stream for as long as it takes them to navigate that extra UI.
>
> I'll post in that defect. Thanks for the response.

[jno...@hirevue.com]

On Thursday, January 7, 2016 at 3:23:39 PM UTC-7, Martin Thomson wrote:
> Think of this not as "can I use your camera", but as "pick a camera
> that I can use". When you pick a file for upload, you don't give the
> site permission to the subsequently upload all the other files in that
> directory. The "remember my choice" option is what is special here.

I do not accept this comparison.

Allowing unfettered access to a file system is simply not equivalent to saying a site has access to all available cameras and microphones. The first is an obviously massive security breach, and nobody here is going to debate adding a setting to firefox that says "can I use your filesystem?" No reasonable person is going to agree that is secure/private.

But the second is completely different. What's the security/privacy impact here?

1. For multiple microphones, we're talking about microphones that are in close proximity of one another, so it isn't like one microphone is going to pick up dramatically different information from another microphone. I do not understand the privacy concern; so long as access to one has been granted, might as well grant access to all of them.

2. For multiple cameras, most users don't even have this. If they do, it's unlikely they point at dramatically different things, except in the case of a cell phone, but that's more of a configuration issue than a privacy/security one. If a user does have a camera intentionally pointing at different things (which is definitely not typical), the user is probably aware of this and they'll take matters into their own hands.

...I think the way Chrome handles this is perfectly acceptable, and I do not understand why Firefox need to behave differently in this regard. Chrome is even MORE lax than what we're talking about; if you go to a page loaded vi HTTPS and allow camera/mic access, that is interpreted as a permanent "allow" to all cameras/microphones. What firefox is doing here is frankly overly restrictive, and for no good reason I can see.

[Jan-Ivar Bruaroey]

On 1/7/16 12:38 PM, jno...@hirevue.com wrote:
> On Thursday, January 7, 2016 at 9:22:15 AM UTC-7, Jan-Ivar Bruaroey wrote:
>> Look closely at the Firefox permission dialog here, and you'll see it
>> doesn't actually let users change the camera or microphone (because the
>> site uses `exact` constraints).
>
> Interesting--is that "exact" constraint documented somewhere?

See [1] and [2].

> Does Chrome support it as well?

Only if you use adapter.js, the official WebRTC polyfill. [3]

That's because Chrome still uses an older `mandatory`/`optional`
constraints syntax, which can express the same thing (`exact` has the
same effect as `mandatory`).

..: Jan-Ivar :.

[1]
https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#Parameters
[2] http://w3c.github.io/mediacapture-main/getusermedia.html#methods-6
[3] https://github.com/webrtc/adapter/

[jno...@hirevue.com]

Thanks Jan-Ivar. I updated our client-side code to use adapter.js, and that was a nice improvement.

On Friday, January 8, 2016 at 4:57:55 PM UTC-7, Jan-Ivar Bruaroey wrote: