Intent to Ship: MediaDevices devicechange event

[Guido Urdaneta]

Contact emails

gui...@chromium.org, h...@chromium.org

Spec

https://w3c.github.io/mediacapture-main/#event-mediadevices-devicechange

Tag review:https://github.com/w3ctag/spec-reviews/issues/57

Summary

This intent covers the addition of the devicechange event to the MediaDevices object. This event is fired when a media device is connected to or removed from the system.

Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/d/msg/blink-dev/wdSV0zUuvJQ/Jd7p_mAIDgAJ

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

The feature has full support on Windows, Linux and ChromeOS.

Mac is supported too, but events corresponding to output-only audio devices currently do not trigger the devicechange event. The reason is that the Mac APIs Chromium currently uses for monitoring support only media input devices. We consider this a bug that may be fixed in the future using other Mac APIs.

There is no support for Android or Android WebView. The reason is that there is no underlying hardware monitoring support in Chromium for these platforms.

Interoperability and Compatibility Risk

The interoperability risk is low since the feature has been in the spec without any significant change for more than two years and is supported by Edge .  The feature is in active development in firefox.

There are no significant compatibility risks associated with this feature as it does not affect existing applications that are not aware of this event.

OWP launch tracking bug

crbug.com/585096

Entry on the feature dashboard

https://www.chromestatus.com/feature/5662847321243648

[Philip Jägenstedt]

This event fires when navigator.mediaDevices.enumerateDevices() would resolve with something different, removing the need for polling, which is great.

A strange thing that Guido and I have discussed is that enumerateDevices() works even without any permission and in that case omits the labels of the devices, but it's still possible to poll the API in that scenario and observe changes. However, the devicechange event wouldn't fire. There are two obvious ways of fixing this inconsistency:

• Fire the devicechange even without any permission, since it could be observed by polling anyway,
  
• Change enumerateDevices() to reject or resolve with dummy data when there's no permission.
  

hta@, thoughts?

[Harald Alvestrand]

This was discussed in the WG.

Firing the devicechange event increases the fingerprinting surface - you can observe the event from multiple tabs, which allows to detect that tabs are running on the same device. The decision was made in the WG that putting this behind the same protection as labels seemed to be a reasonable thing to do.

Spec link: https://w3c.github.io/mediacapture-main/getusermedia.html#mediadevices

[Philip Jägenstedt]

Does the event reveal anything that continuously polling enumerateDevices() would not?

[Philip Jägenstedt]

I should also say that changing enumerateDevices() seems like the better option here. When looking into getSources I noticed its use for (I think) fingerprinting in https://tracker-ssl.beelert.com/tracker.js and presumably enumerateDevices() could be used in the same way, and perhaps already is.

To go down this path would mean no changes to the devicechange event, but it would reopen the spec discussion around enumerateDevices().

[Harald Alvestrand]

I don't quite see it ..... What would be better with changing EnumerateDevices?

"no changes to the devicechange event" - do you mean that we implement it per spec including only firing it when we have the mediainfo permission?

"Continuous" polling has lower time resolution and higher resource load than listening for the event, I think.

[Philip Jägenstedt]

Guido tells me that the devicechange event as implemented already only fires when the permission associated with getUserMedia() is granted.

My concern is how this conversion might end:

DevRel: Here's a new devicechange event, when it fires you can call enumerateDevices() to find out what changed.

Developer: That's great, so I never need to poll enumerateDevices() for changes?

DevRel: Well, the event doesn't fire before the user has already granted permission for getUserMedia().

Developer: OK, but I can still poll enumerateDevices() to see the changes?

DevRel: Yes, but please don't, polling is less accurate and drains the battery.

Developer: ???

If the answer isn't simply "no more polling" then we need a good explanation, and maybe there is one. A small difference in timing seems unlikely to thwart cross-tab tracking where one tab may already have throttled scripts, and as long as it doesn't crash the drivers the resource usage doesn't seem like it would matter to a tracker on desktop.

If enumerateDevices() would simply reject then the issue goes away, but are there good use cases for calling enumerateDevices() before you have permission?

[Harald Alvestrand]

Re enumerate before permission:

The example that's most often come up in discussion is to figure out if the (special) camera you were configured to use the last time is still there.

Runner-up is to figure out whether there are video devices present, and if not, present a pure audio interface.

One possibility that came up at TPAC when discussing the media-info permission was that we could pass a new parameter to enumerateDevices that says "I really need the labels, even if you have to ask the user for permission to get them". This would be a good answer to "what do I do instead of polling".

but that wasn't cited as an immediate suggested change.

[Philip Jägenstedt]

On Mon, Sep 26, 2016 at 2:25 PM Harald Alvestrand <h...@google.com> wrote:

Re enumerate before permission:

The example that's most often come up in discussion is to figure out if the (special) camera you were configured to use the last time is still there.

In this case one should still have the permission from last time, right? If the permission was revoked, then should that not be treated similarly to what the spec suggests for clearing cookies?

Runner-up is to figure out whether there are video devices present, and if not, present a pure audio interface.

Makes sense, do you know of any site that does this? If there is a UI difference based on camera presence, then this seems like a case where reacting to device changes would be useful, and they'll still need to poll enumerateDevices().

One possibility that came up at TPAC when discussing the media-info permission was that we could pass a new parameter to enumerateDevices that says "I really need the labels, even if you have to ask the user for permission to get them". This would be a good answer to "what do I do instead of polling".

but that wasn't cited as an immediate suggested change.

In spec land, is the idea that there should ultimately be separate or merged permissions for enumerating and using? https://w3c.github.io/permissions/#media-devices only mentions getUserMedia(), but has enumerateDevices() in an example...

[Harald Alvestrand]

On Mon, Sep 26, 2016 at 4:29 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

On Mon, Sep 26, 2016 at 2:25 PM Harald Alvestrand <h...@google.com> wrote:

Re enumerate before permission:

The example that's most often come up in discussion is to figure out if the (special) camera you were configured to use the last time is still there.

In this case one should still have the permission from last time, right? If the permission was revoked, then should that not be treated similarly to what the spec suggests for clearing cookies?

Only if it was stored. Firefox doesn't store by default.

 

Runner-up is to figure out whether there are video devices present, and if not, present a pure audio interface.

Makes sense, do you know of any site that does this? If there is a UI difference based on camera presence, then this seems like a case where reacting to device changes would be useful, and they'll still need to poll enumerateDevices().

Cisco has been the most vocal about this, so I suspect they want WebEx to do it.

They also want this to be done before asking for devices; once they've grabbed the mike, they're home safe.

 

One possibility that came up at TPAC when discussing the media-info permission was that we could pass a new parameter to enumerateDevices that says "I really need the labels, even if you have to ask the user for permission to get them". This would be a good answer to "what do I do instead of polling".

but that wasn't cited as an immediate suggested change.

In spec land, is the idea that there should ultimately be separate or merged permissions for enumerating and using? https://w3c.github.io/permissions/#media-devices only mentions getUserMedia(), but has enumerateDevices() in an example...

I specced "device-info" as a separate permission. It ended up as the last line of the section.

[PhistucK]

On Mon, Sep 26, 2016 at 12:15 PM, 'Harald Alvestrand' via blink-dev <blin...@chromium.org> wrote:

you can observe the event from multiple tabs, which allows to detect that tabs are running on the same device

But this is possible with BroadcastChannel anyway (it is even kind of its point, in a way)​ and in others ways (service workers, localStorage and its events, setting and polling for cookies), which do not even react to external events, so I am not sure what this is preventing that is not fully standardized and acceptable?

☆PhistucK

[Philip Jägenstedt]

On Mon, Sep 26, 2016 at 4:55 PM 'Harald Alvestrand' via blink-dev <blin...@chromium.org> wrote:

On Mon, Sep 26, 2016 at 4:29 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

On Mon, Sep 26, 2016 at 2:25 PM Harald Alvestrand <h...@google.com> wrote:

Re enumerate before permission:

The example that's most often come up in discussion is to figure out if the (special) camera you were configured to use the last time is still there.

In this case one should still have the permission from last time, right? If the permission was revoked, then should that not be treated similarly to what the spec suggests for clearing cookies?

Only if it was stored. Firefox doesn't store by default.

I see. In principle it still doesn't require allowing enumeration the first time you visit a site, e.g., there could be a separate permission for enumeration that's set implicitly when getUserMedia() succeeds and which is persistent.

Runner-up is to figure out whether there are video devices present, and if not, present a pure audio interface.

Makes sense, do you know of any site that does this? If there is a UI difference based on camera presence, then this seems like a case where reacting to device changes would be useful, and they'll still need to poll enumerateDevices().

Cisco has been the most vocal about this, so I suspect they want WebEx to do it.

They also want this to be done before asking for devices; once they've grabbed the mike, they're home safe.

I'm thinking that if the thing behaves differently with a camera present, the user is going to be unimpressed if plugging in the camera doesn't work, if reloading is necessary even though in some other cases the site does react to plugging in a camera.

One possibility that came up at TPAC when discussing the media-info permission was that we could pass a new parameter to enumerateDevices that says "I really need the labels, even if you have to ask the user for permission to get them". This would be a good answer to "what do I do instead of polling".

but that wasn't cited as an immediate suggested change.

In spec land, is the idea that there should ultimately be separate or merged permissions for enumerating and using? https://w3c.github.io/permissions/#media-devices only mentions getUserMedia(), but has enumerateDevices() in an example...

I specced "device-info" as a separate permission. It ended up as the last line of the section.

So there is no discussion about a permission for enumerateDevices() as a whole, apart from the filtering of labels? https://wicg.github.io/feature-policy/#usermedia has a way to disable it, but that doesn't align with any combination of permissions AFAICT.

To bring it back to the original concern, if it's a problem that sites can observe changes in the number of devices with no permission, then enumerateDevices() has that problem. To suppress doesn't seem like it could help against fingerprinting, the timing difference would be tiny.

It would be useful to test Edge's behavior and what Firefox has in development. If they already do what the spec says, then it's probably best to ship that behavior and move on, but if not there may be other options.

Feedback from API owners most welcome.

[Harald Alvestrand]

I have misgivings about revisiting wg decisions in i2s threads- can you file an issue on mediacapture-main for this, so that the spec people see it? The wg explicitly wanted the event to fire only when labels are present, I do not want to ship this api as non-compliant.

[Rick Byers]

On Tue, Sep 27, 2016 at 12:27 PM, 'Harald Alvestrand' via blink-dev <blin...@chromium.org> wrote:

I have misgivings about revisiting wg decisions in i2s threads- can you file an issue on mediacapture-main for this, so that the spec people see it? The wg explicitly wanted the event to fire only when labels are present, I do not want to ship this api as non-compliant.

I agree that questions on the API design should be discussed as spec bugs, so yes let's move this discussion there.

 

In terms of interop risk though I think the key question is what implementations for this are already doing.  Guido / Harald, does Edge have this specific behavior?  Or is this possibly an area where the spec doesn't match existing shipping implementations?

Thanks,

   Rick

[Harald Alvestrand]

I'll ask my Microsoft contact.

[Philip Jägenstedt]

Guido tested Edge (on physical hardware) today and also found an existing spec issue:

https://github.com/w3c/mediacapture-main/issues/402

As mentioned there, Edge does fire the event.

When I raised this issue I was hoping that enumerateDevices() could be restricted, but now I'm inclined to say we should match Edge, ask to have the spec changed, and try to address any fingerprinting issues with changes to enumerateDevices(), which would naturally also limit when the events are fired.

[Rick Byers]

That plan sounds great to me!  Thanks for the research Guido and the follow-up Phillip!

In cases of conflict between all existing shipping implementations and the spec (as captured by an outstanding spec issue), I'd personally prefer we either match existing implementations or (if the team who owns the feature believes that's the wrong choice) block shipping on getting the spec issue resolved.  In general, interop (consistency of behavior) tends to matter to web developers more than spec conformance does :-)

Rick

[Rick Byers]

By the way, regarding the fingerprinting concern of this API.  The chromium project has a policy of treating such issues WontFix as fingerprinting is shown to be easy in practice already without any realistic solutions.

[Harald Alvestrand]

Is the recommendation of the Chromium API owners that we should ship with Edge-compliant event firing instead of spec-compliant event firing, then?

It should be easy enough to remove the check.

[Rick Byers]

It looks like there's been some good discussion on the issue.  Ideally the WG will reach a conclusion that the implementors say they're happy to support.

But we don't necessarily need to block shipping on that.  Ignoring the spec for a minute, which behavior does the team who owns the implementation in blink (Guido?)  feel is a better design?  If the blink implementors feel the Edge implementation is the better design, then that's lower interop risk and so I'd personally support them in their desire to ship that (leaving the mismatch as an open spec issue for now).

Rick

[Guido Urdaneta]

Removing the permission check is  easy and it can provide other practical advantages such as unifying the device notification mechanism for PPAPI, which is currently separate and does not check for permissions.

Should we wait until the spec discussion is finished, or is it OK to remove the permission check and ship with interoperability with Edge?

[Philip Jägenstedt]

Harald, how long might it take for the WG to resolve an issue like this? If you (Guido and Harald) would like to match Edge and ship before the spec has changed, I think that's OK if we can reasonably expect the WG to end up agreeing.

[PhistucK]

So can portable native client binaries (while not an open web platform feature, can be loaded without installation in Chrome) do things - for which open web platform features need permissions - without permissions?

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Guido Urdaneta]

Not sure I understand your question, but PPAPI uses enumerations (which do not check for permissions to produce results) to learn about device changes, while the blink devicechange event is fired only if the origin has permission.

My previous comment referred to the fact that if the devicechange event did not require permission checking, the same mechanism could be used by PPAPI too.

[PhistucK]

Cool. Thank you for the clarification.

☆PhistucK

[gui...@google.com]

The implementation of the devicechange event in Blink has changed and, like Edge, it fires the event even if the origin has no permission to access the devices.

This also makes the event match the way enumerateDevices works.

With regards to the spec, the language has been updated to include firing the event to the document that has focus, although Edge and Chromium fire it for all documents listening to the event regardless of focus (tested on Edge 39.14959.1000)

The language of the spec is not 100% clear, so there are a number of spec bugs open related to this: 

- Polling enumerateDevices potentially being a fingerprint

- Need rules for devicechange event when devices are removed. 

- Devicechange events when not focus - permitted or forbidden?

Should we ship now that Blink is interoperable with Edge or should we wait until all the spec issues are resolved?

☆PhistucK

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

Thanks Guido for testing on Edge, that's really important to guide us on issues like this.

https://w3c.github.io/mediacapture-main/ isn't auto-updated to reflect the latest changes, so everyone take care to look at http://rawgit.com/w3c/mediacapture-main/master/getusermedia.html for the time being.

The spec now has this as the condition for firing the event: 'When new media input and/or output devices are made available, if either the permission state of the "device-info" permission is "granted", or any of the local devices are attached to an active MediaStream in the current browsing context, or the active document in the current browsing context is fully active and has focus.'

What you'd like to ship ignores the conditions, and I personally think this is the right call. However, to make sure we end up in a good place, maybe:

• Ask for Chrome security/privacy feedback on this specific point, to check that they are also not concerned.
• Write a PR for the spec for a change that would allow the behavior of Edge and what you'd like to ship.
• Comment on the other relevant issues what you intend to ship, so that nobody is surprised.

With that, if there's still disagreement in the WG and nothing obvious that could be done to resolve it, then I think we should go ahead and ship.

[Martin Šrámek]

Chiming in from from Chrome Privacy side!

Sorry for my delayed reply; I spent a while thinking about this.

I agree with Philip's assessment that the equivalent can be done by polling. The event is relatively rare (plugging/unplugging my headphones does not register as a device change, because the sound card is the device; I only occasionally manipulate my USB webcam), so if it is fired for two different sessions within a second, that's already a very strong signal that it's the same user. Polling once per second does show up on the task manager, but the CPU load would end up being negligible on any JS-heavy site.

So I think that hiding `devicechange` behind a permission does not offer a strong protection. To be consistent with other vendors, and with the fact that `enumerateDevices()` is available without a permission, I think it makes sense to ship without a permission.

On the other hand, the fact that there is no privacy delta between `enumerateDevices()` and `devicechange` means that `enumerateDevices()` is the dangerous one. Not because of fingerprinting by device IDs (those are different for Incognito and after you delete cookies), but because plugging my webcam can tie my regular and incognito sessions of the same site together.

We could hide the new device for a small amount of time from Incognito, i.e. neither `devicechange` nor `enumerateDevices()` would report it. I would argue that this won't break anything, because the valid usecases for "`devicechange` without permission" that I read about in the past discussions are unlikely to happen in two separate sessions at the same time. But as I mentioned before, this would have to be orders of magnitude longer than a second. Maybe wait for focus in incognito instead? The behavior cannot be too different either, because it will reveal the incognito session directly. I'll have to think about this more, and I trust that the WG does too :)

But again, that's not a blocker from my side to ship; that's a suggestion to revisit enumerateDevices(), or any other "give me system configuration that changes occasionally" calls. I think this is interesting to discuss with the Bluetooth API owners as well.

Best regards,

Martin

[Reilly Grant]

We could avoid making the incognito session behavior consistently different by always delivering the event for the active session first and inactive sessions after the delay (and immediately if they are focused).

Are "give me system configuration that changes occasionally" calls more or less dangerous than "give me system state that changes frequently" calls like DeviceOrientationEvent? Both by design have to provide the same data to regular and incognito sessions.

[Harald Alvestrand]

The bug that Jan-Ivar filed on enumerateDevices being equivalently dangerous (or not) with the devicechange event is here:

https://github.com/w3c/mediacapture-main/issues/403

The original bug about the devicechange as fingerprint signal was #402, but we closed that after landing the current text.

[gui...@google.com]

The spec has changed so that the result of enumerateDevices is cached in a browsing session and never updated by device changes unless the user has permission.

Therefore, a page without device permissions can no longer poll enumerateDevices to detect device changes.

We can modify Chromium's implementation of both enumerateDevices and devicechange so that it matches the new spec, although it would break interoperability with Edge

In order to finally ship this feature, should we aim for compliance with the spec or interoperability with Edge's current behavior?

<div class="m_3083624950843734075m_6934146904487140413gmail-m_-884175159304806755gmail_msg m_308362495084373

[PhistucK]

Is Edge going to implement the change as well?

☆PhistucK

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Philip Jägenstedt]

The risk of later masking changes in enumerateDevices an thus withholding devicechange events should be low, but even so I don't think that changing enumerateDevices need block shipping the devicechange event. Can you file a follow-up bug to track that it is eventually either implemented or reverted from the spec?

LGTM1 to ship the devicechange event as currently implemented.

[gui...@google.com]

Filed http://crbug.com/668132.

[gui...@google.com]

Any comments from other API OWNERS?

On Wednesday, November 23, 2016 at 11:23:33 AM UTC+1, Philip Jägenstedt wrote:

[Chris Harrelson]

LGTM2

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Guido Urdaneta]

Hi API OWNERS,

This needs one more LGTM. Any takers?

[Jochen Eisinger]

lgtm3

LGTM2

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.