Intent to Experiment: getCurrentBrowsingContextMedia()

[Elad Alon]

Contact emails

elad...@chromium.org, h...@chromium.org, gui...@chromium.org, hu...@google.com, agp...@chromium.org

Explainer

bit.ly/3nnH9yL

Specification

https://github.com/w3c/mediacapture-screen-share/pull/148

However, this feature is under active discussion in the WebRTC WG. The main point of contention is the security measures necessary to be adopted to unlock a confirmation-only flow. We intend to ship an explicit-selection flow first, which will be the fallback behavior for applications that do not fulfil the security requirements for the confirmation-only flow. Once the security measures of the confirmation-only flow are decided, specified and implemented, we will ship the confirmation-only flow, too.

Summary

We introduce a new API - navigator.mediaDevices.getCurrentBrowsingContextMedia(). This API allows capturing a MediaStream with the current tab's video (and potentially audio), similar to getDisplayMedia(). Unlike getDisplayMedia(), calling this new function will (eventually) present the user with a simple accept/reject dialog. If the user accepts, the current tab will be captured. In all other ways, getCurrentBrowsingContextMedia() will be identical to getDisplayMedia().

However, this feature is under active discussion in the WebRTC WG. The main point of contention is the security measures necessary to be adopted to unlock a confirmation-only flow. We intend to ship an explicit-selection flow first, which will be the fallback behavior for applications that do not fulfil the security requirements for the confirmation-only flow. Once the security measures of the confirmation-only flow are decided, specified and implemented, we will ship the confirmation-only flow, too.

Blink component

Blink>GetDisplayMedia

Search tags

getDisplayMedia, capture, getCurrentBrowsingContextMedia, share, sharing

TAG review

TAG review

TAG review status

Pending

Risks

Interoperability and Compatibility

Gecko: Positive signals for the feature. Initial disagreement over the necessary security measures, but we (Chrome) are coming to see things their way and expect to come to agreement over something similar to their initial security proposal.

WebKit: No signal

Web developers: Positive signals.

Goals for experimentation

Prove usefulness of the feature to early adopters (e.g. Google Workspace).

Receive feedback over changes necessary to the feature.

Pave way to additional features currently planned for development based on this feature. (E.g. region-capture.)

Reason this experiment is being extended

None

Ongoing technical constraints

As mentioned above at more length, we are currently only experimenting over the explicit-selection flow. The confirmation-only flow will be developed at a later time.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No; only desktop platforms for the time being.

Is this feature fully tested by web-platform-tests?

No

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5045313003847680

This intent message was generated by Chrome Platform Status.

[Thomas Steiner]

Hi Elad,

(Sending this as a curious developer, not as an API owner.)

On Sat, Feb 20, 2021 at 5:25 PM Elad Alon <elad...@chromium.org> wrote:

Contact emails

elad...@chromium.org, h...@chromium.org, gui...@chromium.org, hu...@google.com, agp...@chromium.org

Explainer

bit.ly/3nnH9yL

Specification

https://github.com/w3c/mediacapture-screen-share/pull/148

However, this feature is under active discussion in the WebRTC WG. The main point of contention is the security measures necessary to be adopted to unlock a confirmation-only flow. We intend to ship an explicit-selection flow first, which will be the fallback behavior for applications that do not fulfil the security requirements for the confirmation-only flow. Once the security measures of the confirmation-only flow are decided, specified and implemented, we will ship the confirmation-only flow, too.

Summary

We introduce a new API - navigator.mediaDevices.getCurrentBrowsingContextMedia(). This API allows capturing a MediaStream with the current tab's video (and potentially audio), similar to getDisplayMedia(). Unlike getDisplayMedia(), calling this new function will (eventually) present the user with a simple accept/reject dialog. If the user accepts, the current tab will be captured. In all other ways, getCurrentBrowsingContextMedia() will be identical to getDisplayMedia().

However, this feature is under active discussion in the WebRTC WG. The main point of contention is the security measures necessary to be adopted to unlock a confirmation-only flow. We intend to ship an explicit-selection flow first, which will be the fallback behavior for applications that do not fulfil the security requirements for the confirmation-only flow. Once the security measures of the confirmation-only flow are decided, specified and implemented, we will ship the confirmation-only flow, too.

Blink component

Blink>GetDisplayMedia

Search tags

getDisplayMedia, capture, getCurrentBrowsingContextMedia, share, sharing

TAG review

TAG review

TAG review status

Pending

Risks

Interoperability and Compatibility

Gecko: Positive signals for the feature. Initial disagreement over the necessary security measures, but we (Chrome) are coming to see things their way and expect to come to agreement over something similar to their initial security proposal.

Can you provide links to the discussions that were had?

 

WebKit: No signal

Did you ask (https://lists.webkit.org/mailman/listinfo/webkit-dev)?

 

Web developers: Positive signals.

Intuitively the gained simplicity sounds like something users and developers would welcome. Could you link to evidence (https://goo.gle/developer-signals)? 

[Philip Jägenstedt]

This is "just" and Inten to Experiment and not to Ship, but looking at https://github.com/w3c/mediacapture-screen-share/pull/148, defining the lifetime of capture precisely seems important, and also resolving the naming question. That, and spelling out the use cases motivating this for other vendors, seem like the most important things to move this forward to something shipping.

In the experiment you intend to run here, since the new UI isn't implemented, will there be any differences to getDisplayMedia()? If there's no or only minimal differences, what are web developers going to experiment with which will inform the final design?

Best regards,

Philip

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALgRrLkbmjt-oCn3NP9%2Bu-YKZKSDhv%2BHTYj2r87n2q2xhrJ6Ag%40mail.gmail.com.

[Elad Alon]

@Thomas: Sorry for not having answered you yet. I want to respond to the ongoing Github threads, signaling our agreement over things we were blocked on previously, before linking you to those threads. I will do so soon.

@Philip:

IIUC, the life-time of the capture was a question which was sufficiently answered on the thread. Namely, when an application captures the current-tab, navigation of the current tab unloads the capturing application, thereby terminating the capture - so no issue. (And if we default to getDisplayMedia-like behavior, then lifetime is handled as it is in getDisplayMedia.)

For use-cases to other vendors, here are two possible scenarios:

• A browser-based game can capture footage of itself and transmit it to the back-end of a service like Twitch.
• A product similar to Google Slides could capture itself and transmit its own video to a VC service like Zoom / Microsoft Teams / Google Meet.

"resolving naming questions"

Changing the name at a later time would not pose a problem, IMHO. Apps can migrate easily.

"since the new UI isn't implemented"

I am not sure about that? We have two UIs, both implemented, but only one on which we'd like to start experimenting.

1. The confirmation-only UI will be subject to (a) site-isolation and (b) a new header. We're not starting experimentation on it yet. (The UI itself is implemented, but gated behind a constant set to false.)

2. The explicit-selection UI is implemented and gated behind OT (following the CL). We'd like to start experimentation with it.

The difference to getDisplayMedia() is that with the explicit-selection UI of getCurrentBrowsingContextMedia(), the browser is allowed to use a different UI than for gDM, so long as that different UI still follows the specification-imposed requirements of gDM. In our case, this means shuffling the order of sources in a way that favors choosing the current tab. (This is spec-compliant and has been approved by Chrome Security, btw.)

[Philip Jägenstedt]

Thank Elad, LGTM to experiment.

On Wed, Feb 24, 2021 at 12:46 PM Elad Alon <elad...@google.com> wrote:

@Thomas: Sorry for not having answered you yet. I want to respond to the ongoing Github threads, signaling our agreement over things we were blocked on previously, before linking you to those threads. I will do so soon.

@Philip:

IIUC, the life-time of the capture was a question which was sufficiently answered on the thread. Namely, when an application captures the current-tab, navigation of the current tab unloads the capturing application, thereby terminating the capture - so no issue. (And if we default to getDisplayMedia-like behavior, then lifetime is handled as it is in getDisplayMedia.)

Alright, that sounds sensible in the simple case. What's not clear is what the behavior should be when invoking the API in same-origin or cross-origin iframes, if that works at all or what should happen. Note that this is important for being able to fully spec and test the API, but it's doubtful it matters to anyone experimenting with the API.

 

For use-cases to other vendors, here are two possible scenarios:

• A browser-based game can capture footage of itself and transmit it to the back-end of a service like Twitch.
• A product similar to Google Slides could capture itself and transmit its own video to a VC service like Zoom / Microsoft Teams / Google Meet.

Great, capturing those in an explainer would be great.

"resolving naming questions"

Changing the name at a later time would not pose a problem, IMHO. Apps can migrate easily.

Indeed, this doesn't pose much risk for experimenting. Actually resolving the naming question to everyone's satisfaction might still be hard, of course.

"since the new UI isn't implemented"

I am not sure about that? We have two UIs, both implemented, but only one on which we'd like to start experimenting.

1. The confirmation-only UI will be subject to (a) site-isolation and (b) a new header. We're not starting experimentation on it yet. (The UI itself is implemented, but gated behind a constant set to false.)

2. The explicit-selection UI is implemented and gated behind OT (following the CL). We'd like to start experimentation with it.

The difference to getDisplayMedia() is that with the explicit-selection UI of getCurrentBrowsingContextMedia(), the browser is allowed to use a different UI than for gDM, so long as that different UI still follows the specification-imposed requirements of gDM. In our case, this means shuffling the order of sources in a way that favors choosing the current tab. (This is spec-compliant and has been approved by Chrome Security, btw.)

Great, thank you, so there is a difference, I got the impression that the explicit-selection UI was precisely the same as for gDM, which wouldn't have been very useful to experiment with, effectively an alias.

[Elad Alon]

Gecko: Positive signals for the feature. Initial disagreement over the necessary security measures, but we (Chrome) are coming to see things their way and expect to come to agreement over something similar to their initial security proposal.

Can you provide links to the discussions that were had?

General discussion

Security-focused discussion

 

 

WebKit: No signal

Did you ask (https://lists.webkit.org/mailman/listinfo/webkit-dev)?

No, but we did engage with them in the WebRTC WG.

  

Web developers: Positive signals.

Intuitively the gained simplicity sounds like something users and developers would welcome. Could you link to evidence (https://goo.gle/developer-signals)? 

There is an externally-viewable document here with an embedded YouTube video. If you look closely at timestamp 00:38 of that video, you can see a mock that shows synergy between collaboration on a Slides deck and a video-conferencing application that looks like Meet. So that's two interested web-devs.