Intent to Prototype - getCurrentBrowsingContextMedia

Elad Alon

unread,

Oct 9, 2020, 12:43:12 PM10/9/20

to blin...@chromium.org

Contact emails

elad...@google.com

Explainer

1. Explainer.

2. Public discussion on w3 mailing-list.

Design doc/Spec

PR on mediacapture-screen-share spec.

Summary

Provide an API for capturing the tab from which the API is called (as opposed to the existing API which presents the user with a wide selection of capture-sources). See explainer for more details.

Motivation

Sometimes the app is interested in capturing the current tab. Two examples:

1. Presenting a document in the current tab with some remote user (e.g. sharing slides to a VC).

2. Recording footage of a locally played game to a video file.

By providing an API for capturing this tab to the application, the user can be presented with a simpler prompt (yes/no), which would prevent the user from accidentally sharing too much (e.g. another tab).

Risks

Interoperability and Compatibility

Other browsers might not implement. This is not a risk at this stage, as we're only prototyping.

This feature cannot break compatibility with existing applications.

Ergonomics
This will often be used in conjunction with PeerConnections.
Like getDisplayMedia(), after which this API is modelled, the user will be presented with a prompt, and further interaction with the page will be blocked until the user responds to the prompt.

Activation

The feature is simple to use and will be documented in the spec before shipping.

Debuggability

Not applicable.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Only on non-mobile platforms, as is the case with getDisplayMedia().

Link to entry on the feature dashboard

Link

Requesting approval to ship?

No

Łukasz Polowczyk

unread,

Oct 9, 2020, 7:02:07 PM10/9/20

to blink-dev

Next step: `HTMLElement.prototype.getElementMedia()`
Why not?

PhistucK

unread,

Oct 10, 2020, 6:01:45 AM10/10/20

to Elad Alon, blink-dev

One more use case is for things like "Send feedback". Currently, websites use various libraries that try to recreate the page as an image. This could make them send the exact representation of the page, including iFrames (in conjunction with the image capture API, of course), using a pretty contextual prompt.

(I know this feedback is useless in Google products, but it will simplify and enhance its implementation greatly :P)

Of course, since getDisplayMedia already does all of that, this use case is supported there, but the prompt is more contextual. I am not sure why the getDisplayMedia specification tells the browser not to limit the user choice according to the constraints, but whatever...

(It seems like adhering to the constraints is a needed feature, as Google Meet successfully does that and I am guessing this it is only happening in Chrome due to some unfortunate and unfair special casing)

Sounds like it can be solved with getDisplayMedia by automatically picking the current tab as the default (using a constraint), among all of the other available choices.

I would say this is an overkill. getDisplayMedia should be revised to easily accommodate this.

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMO6jDP%3DeOmKpXha6GUm0m9rggo4x12DiT_NAbkvPzWz4%3DQG6w%40mail.gmail.com.

Yoav Weiss

unread,

Oct 12, 2020, 4:06:31 AM10/12/20

to Elad Alon, blink-dev

On Fri, Oct 9, 2020 at 6:43 PM 'Elad Alon' via blink-dev <blin...@chromium.org> wrote:

Contact emails
elad...@google.com

Explainer
1. Explainer.
2. Public discussion on w3 mailing-list.

Design doc/Spec
PR on mediacapture-screen-share spec.

Might be good to kick off an early TAG review.

Summary
Provide an API for capturing the tab from which the API is called (as opposed to the existing API which presents the user with a wide selection of capture-sources). See explainer for more details.

Motivation
Sometimes the app is interested in capturing the current tab. Two examples:
1. Presenting a document in the current tab with some remote user (e.g. sharing slides to a VC).
2. Recording footage of a locally played game to a video file.
By providing an API for capturing this tab to the application, the user can be presented with a simpler prompt (yes/no), which would prevent the user from accidentally sharing too much (e.g. another tab).

Risks
Interoperability and Compatibility
Other browsers might not implement. This is not a risk at this stage, as we're only prototyping.

Have you solicited feedback from other vendors?

Have they chimed in on the WebRTC mailing list thread?

This feature cannot break compatibility with existing applications.

Ergonomics
This will often be used in conjunction with PeerConnections.
Like getDisplayMedia(), after which this API is modelled, the user will be presented with a prompt, and further interaction with the page will be blocked until the user responds to the prompt.

Activation
The feature is simple to use and will be documented in the spec before shipping.

Debuggability
Not applicable.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
Only on non-mobile platforms, as is the case with getDisplayMedia().

Link to entry on the feature dashboard
Link

Requesting approval to ship?
No

--

Harald Alvestrand

unread,

Oct 12, 2020, 5:01:19 AM10/12/20

to Łukasz Polowczyk, blink-dev

HTMLElement.getElementMedia(): Because we can't make it work?

Seriously .... there's a place in the stack that assembles everything that goes in a tab onto a surface that we then can make a capture of.

There's no place in the stack that assembles everything that goes on a HTML element (and nothing else) onto a surface. (Consider, for instance, the process of deciding whether CSS on containing elements needs to be applied to the captured element or not).

Yes, we considered it before making this proposal. But we decided not to pursue it.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/23bbf831-bdca-463d-a678-91323205624do%40chromium.org.

Harald Alvestrand

unread,

Oct 12, 2020, 5:04:33 AM10/12/20

to PhistucK, Elad Alon, blink-dev

getBrowserContextMedia() + mediacapture-image should make this trivial... we have also considered adding features that will constrain the capture to a certain region of the tab (something "send feedback" would definitely want).

But given that getBrowserContextMedia() has the same issues with the Same Origin Policy as getDisplayMedia(), I think we'll always have a permission prompt or equivalent - it won't be as smooth as Android's screen-capture functionality.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BbF91HFeLBRGvVeBWrU8pcbRY-c8fZZ%2B5jDHfi6JgOVw%40mail.gmail.com.

Philipp Hancke

unread,

Oct 12, 2020, 6:30:56 AM10/12/20

to Harald Alvestrand, PhistucK, Elad Alon, blink-dev

Note that the third gif in https://cloud.google.com/blog/products/workspace/introducing-google-workspace doesn't show any security UI. I hope that is made up...

I also hope this is not using the thunk extension that Hangouts continues to rely on (see https://bugs.chromium.org/p/chromium/issues/detail?id=850536)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVHKoWTqvCzTZJ9APC0ocU_%2B6pc_VCes_P2rV7pcgR2h_Q%40mail.gmail.com.

Elad Alon

unread,

Oct 12, 2020, 11:49:48 AM10/12/20

to Yoav Weiss, blink-dev, Harald Alvestrand, Guido Urdaneta, Palak Agarwal

Other vendors - (1) Jan-Ivar Bruaroey from Mozilla and (2) Youenn Fablet from Apple have commented on the public discussion. The topic is set to be discussed on TPAC2020; it will be added to the agenda soon.

TAG review - perhaps best done after the result of the discussion at TPAC. The session is on October 22.

Elad Alon

unread,

Oct 12, 2020, 11:49:48 AM10/12/20

to PhistucK, blink-dev

Revising getDisplayMedia was indeed my immediate initial thought, but as you've noted yourself, historical decisions were made to disallow that approach. The change to the spec to allow either (1) a variant of getDisplayMedia, or (2) a revision of getDisplayMedia, will be discussed in TPAC. Until then, I will be prototyping the approach which interferes less with the existing API, requires less approvals, and is generally less likely to introduce security regressions (because the entire thing will be cordoned off by a flag).

On Sat, Oct 10, 2020 at 12:01 PM PhistucK <phis...@gmail.com> wrote:

Reply all

Reply to author

Forward