Intent to Implement: Disable Local Playback During Audio Sharing

[Qiang Chen]

Contact emails

qian...@chromium.org, m...@chromium.org, nik...@chromium.org

Spec

Link to spec. You should have at least an unofficial spec in hand and/or have discussed the API with other browser vendors or standards bodies before sending an intent to implement. If your change is not yet at this stage of maturity, feel free to solicit feedback informally on blink-dev instead.

https://github.com/w3c/mediacapture-screen-share/issues/43

Summary

Add an audio boolean constraint “googDisableLocalEcho” for getUserMedia API. It only works with desktop or tab sharing. When setting this to true, we will mute the local playback for the shared audio.

Motivation

We can anticipate both use cases with muting or unmuting the local playback of the shared audio.

For example, for Google Cast, we’d expect to mute the local playback, because in general the receiver is in the same room with the sender, and thus both ends playing sound out would be annoying. However, for hangouts video call, when the end user is sharing some audio (maybe some YouTube music), we’d expect that both peers can enjoy the audio.

Current default setting is that for source which is selected through picker window, we unmute the local playback, otherwise mute. Because, we anticipated that Hangouts would probably use the picker window to pick a source to share, and Google Cast would not. However, technically there is no such restriction, and we cannot predict how the following Javascript would use a stream. So we need somehow a way Javascript would tell C++ code which way should we go.

Interoperability and Compatibility Risk

I do not think there is an issue, as we keep the default setting unchanged. Namely, the developer could choose not to specifying this constraint, and we will use the current default setting.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No. Not on android, because desktop capture is not supported on android, and this is just a parameter for audio part of desktop capturing.

OWP launch tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=647801

Link to entry on the feature dashboard

https://www.chromestatus.com/features/5056629556903936

Requesting approval to ship?

Yes.

[PhistucK]

On Wed, Sep 28, 2016 at 12:45 AM, 'Qiang Chen' via blink-dev <blin...@chromium.org> wrote:

googDisableLocalEcho

No. New prefixed features are explicitly banned by the Blink policy.

☆PhistucK

[Philip Jägenstedt]

Qiang, the prefix issue is easily fixed by not prefixing, but can you share a snippet of code for context? There's no web exposed API for screen/window/tab sharing, so I guess this would actually be called from something in the Cast UI code?

[PhistucK]

It is web exposed, but only to Google Hangouts (not through extensions or anything, a regular JavaScript in a regular webpage, just on certain origins). :(

For other websites, there is a command line flag that activates screen, tab and window sharing.

And if I am not mistaken, it is called chromeScreen or something else that is prefixed with chrome.

☆PhistucK

[Rick Byers]

On Tue, Sep 27, 2016 at 5:45 PM, 'Qiang Chen' via blink-dev <blin...@chromium.org> wrote:

Contact emails

qian...@chromium.org, m...@chromium.org, nik...@chromium.org

Spec

Link to spec. You should have at least an unofficial spec in hand and/or have discussed the API with other browser vendors or standards bodies before sending an intent to implement. If your change is not yet at this stage of maturity, feel free to solicit feedback informally on blink-dev instead.

https://github.com/w3c/mediacapture-screen-share/issues/43

Summary

Add an audio boolean constraint “googDisableLocalEcho” for getUserMedia API. It only works with desktop or tab sharing. When setting this to true, we will mute the local playback for the shared audio.

Motivation

We can anticipate both use cases with muting or unmuting the local playback of the shared audio.

For example, for Google Cast, we’d expect to mute the local playback, because in general the receiver is in the same room with the sender, and thus both ends playing sound out would be annoying. However, for hangouts video call, when the end user is sharing some audio (maybe some YouTube music), we’d expect that both peers can enjoy the audio.

Current default setting is that for source which is selected through picker window, we unmute the local playback, otherwise mute. Because, we anticipated that Hangouts would probably use the picker window to pick a source to share, and Google Cast would not. However, technically there is no such restriction, and we cannot predict how the following Javascript would use a stream. So we need somehow a way Javascript would tell C++ code which way should we go.

Interoperability and Compatibility Risk

I do not think there is an issue, as we keep the default setting unchanged. Namely, the developer could choose not to specifying this constraint, and we will use the current default setting.

The main point of this request for new features is to describe how likely it is that this API will eventually be widely adopted by all browsers (that's the "interop risk").   Given that there's no spec yet (and no signals from other browser vendors?), I'd say the interop risk is high.  That's OK at "intent to implement time", but for an "intent to ship" we'd want to see that risk brought down.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No. Not on android, because desktop capture is not supported on android, and this is just a parameter for audio part of desktop capturing.

OWP launch tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=647801

Link to entry on the feature dashboard

https://www.chromestatus.com/features/5056629556903936

Requesting approval to ship?

Yes.

This subject / template is just for "implement", not "ship".  Given the above, discussion of shipping seems premature.  Do you agree?

[Qiang Chen]

Let me clarify it.

It is web exposed, and NOT only to Hangouts. Other website can access it through their extension. 

desktopCapture.chooseDesktopMedia({"screen" ,"window", "tab", "audio"}, handler_callback) ;

Above function call will launch a so-called picker window, and the end user could choose a source to share, and could control whether he wants to share audio or not, and it will generate an ID for the picked source.

Then on JS side, we need to call

getUserMedia({

   video: {mandatory:{

        chromeMediaSource: "desktop",

        chromeMediaSourceId: <The Id You Get From chooseDesktopMedia>

   }},

   audio:{ mandatory:{

        chromeMediaSource: "desktop",

        chromeMediaSourceId: <The Id You Get From chooseDesktopMedia>

   }}

})

to get the stream.

Now we want to add a key in the audio constraint to control whether we should mute or unmute the local playback of the shared audio. Like this

getUserMedia({

   video: {mandatory:{

        chromeMediaSource: "desktop",

        chromeMediaSourceId: <The Id You Get From chooseDesktopMedia>

   }},

   audio:{ mandatory:{

        chromeMediaSource: "desktop",

        chromeMediaSourceId: <The Id You Get From chooseDesktopMedia>,

        disableLocalEcho: true

   }}

})

[Qiang Chen]

The main point of this request for new features is to describe how likely it is that this API will eventually be widely adopted by all browsers (that's the "interop risk").   Given that there's no spec yet (and no signals from other browser vendors?), I'd say the interop risk is high.  That's OK at "intent to implement time", but for an "intent to ship" we'd want to see that risk brought down.

Got it, is there anything we need to do now?

This subject / template is just for "implement", not "ship".  Given the above, discussion of shipping seems premature.  Do you agree?

OK, then let's just discuss implement.

[Rick Byers]

On Wed, Sep 28, 2016 at 2:22 PM, 'Qiang Chen' via blink-dev <blin...@chromium.org> wrote:

The main point of this request for new features is to describe how likely it is that this API will eventually be widely adopted by all browsers (that's the "interop risk").   Given that there's no spec yet (and no signals from other browser vendors?), I'd say the interop risk is high.  That's OK at "intent to implement time", but for an "intent to ship" we'd want to see that risk brought down.

Got it, is there anything we need to do now?

I think the main thing is to drive discussion with the other vendors (presumably starting with the issue you filed) to reach a consensus on what precisely this API should look like, and get the result captured in some spec somewhere (presumably, but not necessarily, mediacapture-screen-share).  

[PhistucK]

Wow, that is so, so bad. :(

☆PhistucK

[Rick Byers]

So if I understand correctly, it's a Chrome extension API that let's an extension initiate screen capture and a non-standard extension to getUserMedia that gives the page access to that stream.  

Definitely we should be working to move this to standardized APIs (mediacature-screen-share I asume), but as for our discussions of phasing out webkit-prefixed APIs I personally think it's reasonable to be pragmatic here in enabling incremental progress towards more standard / interoperable APIs.  Thank you Qiang for proposing this new bit as a standard API instead of further extending the pattern of proprietary chrome APIs plumbed through blink as an opaque dictionary!  There really should be a reasonable low-overhead path for you to do along a path to something that will someday work in all browsers :-)

Separately, is there any work planned on implementing the other parts of the W3C mediacapture-screen-share API in blink?

Rick

[PhistucK]

Tangentially, since they are not web exposed (without extensions, which means you can contact the authors through a internal web store search and announce the change), those prefixed properties really should be removed.

I just realized Chrome has four different prefixes (if I am not missing anything)... webkit, goog, chrome and internal. :(

☆PhistucK

[Qiang Chen]

Actually, I am not familiar with API change process, and how to communicate with other vendors.

niklase@ and hta@, can you take a look at this ITI?

On Wed, Sep 28, 2016 at 12:31 PM, Rick Byers <rby...@chromium.org> wrote:

[Off list for a minute]

The big picture approach to web standards within Google is documented at go/standards-process.  Are you or others at Google already involved in the mediacapture-screen-share spec?  Maybe hta@ is the right one to help you push this API through the standards process?

It's a little weird to ship the 'disableLocalEcho' bit without having shipped standard versions of screen capture APIs, but it's probably OK (similar to discussions we've had about not blocking unprefixing some APIs on unprefixing everything needed to use them).  Is there someone at Google actively working on implementing the rest of the standard screen capture API?  I found this Available bug.

Rick

[Qiang Chen]

If you just mean getUserMedia.

Then from my code reading, I found "chromeMediaSourceType" is interchangeable with "sourceType", and "chromeMediaSourceId" is interchangeable with "sourceId".

And to make it clear:

getUserMedia is not an extension API.

Only chooseDesktopMedia is extension only.

Theoretically, a website could wrap only chooseDesktopMedia call in its extension, and put getUserMedia call on the website javascript.

[PhistucK]

It could be a one-line extension, but it will still not work without it, so it requires an extension which means you can eliminate it faster by contacting the authors. I doubt it spread widely.

☆PhistucK

[Rick Byers]

On Wed, Sep 28, 2016 at 3:51 PM, Qiang Chen <qian...@google.com> wrote:

If you just mean getUserMedia.

Then from my code reading, I found "chromeMediaSourceType" is interchangeable with "sourceType", and "chromeMediaSourceId" is interchangeable with "sourceId".

Are those in some specification somewhere?  I don't see them here.  They're not the same as 'deviceId' and 'groupId', are they?

[Qiang Chen]

Not same with deviceId and groupId. 

They are for desktop (or window or tab) capture only.

I am not sure whether they are in some spec or not.

[Qiang Chen]

Follow up.

Any actionable step I can take?

[Anton Vayvod]

Could you rather pass 'disableLocalEcho' via the extensions 'desktopCapture' API? So the returned stream id passed to 'getUserMedia' would already have this information bound to it. Then you wouldn't have to add any new web exposed parameters to 'getUserMedia' that aren't part of the standard and could avoid this thread altogether.

Otherwise, it feels like you'd have to standardize a whole package of unspecified parameters (or even a new API that achieves the same goal in a WebPlatform-friendly way without extensions) which would be a much longer way to go.

[Rick Byers]

As an "intent to implement" there's no approval needed (see launch process) - you're good to land the code for this in blink behind a RuntimeEnabledFeature marked status=experimental.

To make progress towards having an API we can have confidence in shipping and being locked into, you should work with hta@ on the standards front.

[Elliott Sprehn]

I don't think we should land code with a goog prefix. Please fix that before moving forward here. :)

[Qiang Chen]

This is a little bit complicated, as technically we can do screen or tab capture without launching picker window, namely without calling chooseDesktopMedia.  (Difference is that without launching picker window, you are not able to specify some tab to share, but can only share your own tab)

Thus to cover those cases we need to put the new variable in getUserMedia. And logically, it is more natural to make the parameter in getUserMedia.

I'll take Rick's suggestion to put it first behind a RuntimeEnabledFeature.

Yes, I already removed goog from my code.

Thanks.

[Rick Byers]

Yep that's already been agreed upon in the spec issue.  Let's keep the design discussion there where other vendors can be expected to participate :-)

[PhistucK]

This information is quite disconcerting... So any web page can share the tab currently without alerting the user at all about sharing the screen?

Because if so, that is a security issue.

☆PhistucK

[Qiang Chen]

Nope. It is an extension API called tabCapture, whose signature is the same as getUserMedia, and actually uses the same code path with getUserMedia. So no security issue.

[Daniel Ehrenberg]

You missed v8, as in Intl.v8BreakIterator! I hope Intl.Segmenter can lead this towards deprecation and removal... (https://github.com/littledan/Segmenter).

Dan

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[PhistucK]

Right, it slipped my mind. :S

☆PhistucK

[PhistucK]

That is a relief.

☆PhistucK