New extension API proposal: chrome.tabCapture.capturePresentation

[Yuri Wiitala]

Would appreciate you all taking a look at my proposal: https://docs.google.com/document/d/1oSsCaWRjkDBVYfLN_wQVghPlPwgIsgzp8jJZdxAwcLc

Thanks,

Yuri

[Yuri Wiitala]

If there are no other comments, I'd like to proceed with coding/landing this as a private API.  May I have the LGTM stamps from apps-dev@ and security-enamel@?

Thanks,

Yuri

[Benjamin Kalman]

Well despite conversation on the doc I still don't understand the point of this cookie-less stuff, which is why I gave up commenting. It seems like there are use cases beyond being a private API, like a general offscreen tab API (which once upon a time actually existed). The last comment I heard was that mfoltz was going to explain it to me.

[mark a. foltz]

By the "cookie-less stuff" do you mean the rendering of offscreen tabs in an isolated profile?

The following thread on the Webscreens CG debates the pros and cons of this approach for the Presentation API.  In the end it was decided that this approach would provide the most deterministic behavior for the Web developer, who doesn't know if their presentation is rendered by the same UA or a different one.

https://lists.w3.org/Archives/Public/public-webscreens/2014Aug/0012.html

I am not opposed to providing the extension API an option to render in the same profile as the extension.  But it's not required for us to implement local rendering of presentations.

Feel free to set up some time to discuss further if you'd like!

m.

[Benjamin Kalman]

Yes that's what I meant.

I was struggling with how these general concepts (like a background DOM, or a page in an isolated profile) are being tied so specifically to the presentation API. A background DOM sounds great - like a Worker with a DOM. A page in an isolated profile sounds great - sort of like a sandboxed iframe.

but I do understand that standardising only the intersection of that is a tractable task, while the pieces are likely not.

(incidentally while I understand the reasoning behind an isolated profile, it also seems like hand-holding for developers; why not give a developer the opportunity to adjust their presentation based on whether it's local or not? there is a big difference having access to IndexedDB locally vs having to ship it over a message channel. not a point worth discussing here, I know)

I am also struggling with how the web presentation API relates to this extension API. It was explained to me in the doc that the extension version is used as part of implementing the web's presentation API... but the API surfaces for each look very similar. I think I need this explained to me more slowly.

[mark a. foltz]

On Wed, Jul 22, 2015 at 2:18 PM, Benjamin Kalman <kal...@chromium.org> wrote:

Yes that's what I meant.

I was struggling with how these general concepts (like a background DOM, or a page in an isolated profile) are being tied so specifically to the presentation API. A background DOM sounds great - like a Worker with a DOM. A page in an isolated profile sounds great - sort of like a sandboxed iframe.

but I do understand that standardising only the intersection of that is a tractable task, while the pieces are likely not.

(incidentally while I understand the reasoning behind an isolated profile, it also seems like hand-holding for developers; why not give a developer the opportunity to adjust their presentation based on whether it's local or not? there is a big difference having access to IndexedDB locally vs having to ship it over a message channel. not a point worth discussing here, I know)

I am also struggling with how the web presentation API relates to this extension API. It was explained to me in the doc that the extension version is used as part of implementing the web's presentation API... but the API surfaces for each look very similar. I think I need this explained to me more slowly.

Sure, I can fill you in.  The Presentation API is implemented by a combination of an in-browser component (Media Router) and a component extension (think of it as a greatly simplified and generalized version of the Cast extension).  The component extension uses existing extension APIs to implement features like tab and desktop mirroring and Cast flinging support.  Yuri's proposal extends it to support offscreen rendering and capture of presentations for Presentation API.

A design doc (a bit out of date) is here:

https://www.chromium.org/developers/design-documents/media-router

The section on offscreen rendering for Presentation API has a section marked TODO(miu).  Hint hint :)

[Yuri Wiitala]

Ben,

I want to be clear that our goal here is to implement a private extension API, whitelisted for use only by the Media Router component extension.  Making this API public is currently just an interesting thought experiment.  That said, I do understand the value in the upfront forward-thinking here.

It seems that your primary concern is around having two APIs that look similar and maybe do the same thing.  In fact, the web Presentation API encompasses a much larger superset of functionality compared to the extension API:

• The extension API says nothing about how presentation sessions are formed, live/die, are joined (peer-to-peer), or how messaging to/from the presentation page will occur.  None of this functionality is in the scope of this API.
  
• The extension API is used internally by Chrome only to simulate the rendering of the presentation page on a remote display.  This is in contrast to the web Presentation API, which hides the "where the presentation page is rendered" detail completely from the web developer.  This is why we keep pointing you to the Chrome Media Router design doc, as the MR is what handles web Presentation API requests by looking at the capabilities of a remote display and then deciding whether Chrome needs to launch an off-screen tab to do the web rendering.

If you still have reservations about proceeding, let's have a face-to-face, since we're stretching the limits of e-mail now.  ;-)

Thanks,

Yuri

[Benjamin Kalman]

You're putting the API on a public namespace - if you believe this only makes sense as a private API, then put it on a private namespace. Chromecast already has its own private namespace for things right? However something like this, or components of it, seem like a perfectly reasonable public API. Perhaps we should chat about it in person.

[Yuri Wiitala]

Sounds good.  I just sent a meeting invite for Tuesday.  Calendar fragmentation forced me to make it 1/2 hr.  Hope that's enough time.

[Benjamin Kalman]

lgtm, I chatted to Yuri/Mark/Stephen. The API is sufficiently non-specific to have potential use cases outside of Cast, so we should do the standard dev channel publicly, stable channel cast launch. Gather feedback, monitor usage, etc.

I'm going to make some more comments on the doc to iron out a couple more things, but don't wait for that.

Security team does still need to look at this.

[Yuri Wiitala]

I updated the doc today with everything decided in our F2F with Ben.  The major changes:

1. We are making this a general-purpose offscreen tab API that can either operate in a sandboxed mode (i.e., incognito, but no origin requirements) or a non-sandboxed mode (i.e., use current user profile, extension must allow origin).  I've also changed the name of the API to chrome.tabCapture.captureOffscreenTab() to reflect this new understanding.

2. The new goal is to launch a public API on dev channel.  After a [several quarter] "baking period," taking community/external feedback into account, we will proceed with a stable-channel release.

[Nathan Parker]

Hi Yuri --

Is the plan to still to keep this a private, white-listed API? And does the ability to open a hidden tab change what can be captured/streamed?  The plans for Presentation API were to just allow Cast and g+ hangouts, if I'm not mistaken.  Can third parties capture content via this api?

I've made some comments on the doc.

 -- Nathan

[Yuri Wiitala]

Saw this while going through old e-mails.  As we've discussed on the doc since, the plan is now to provide a public API.  We'll start with dev-channel only (with whitelisting for the built-in Chrome Media Router extension's use case), and proceed iteratively from there.  So, yes, third parties will be able to capture content with this API (on canary/dev channel only, to start).

[Yuri Wiitala]

tl;dr: I'm back.  I'll set up a meeting to continue security discussion. 

Apologies for letting this discussion linger, but I'm back now and able to give it full attention.  ;-)

So, originally, we were proposing this as a private/whitelisted extension API.  And, it's possible that for the short-term we may want to go that route to support the Chrome Media Router for M47.

However, Ben convinced us the feature would be useful as a public extension API and I'd also like to proceed on that front for the long-term.  I think the high-level plan there is to start with the public API enabled in dev-channel only and, after it has baked for a few Chrome milestones, launch it for stable-channel.

At this point, there are some lingering issues on the security side of the discussion.  I feel I am not fully qualified to vet all of these out on my own, or even respond to some without more context.  So, I'd like to meet on VC to talk about them.  In particular, I'd like to:

1. Make sure we're already good-to-go for private extension API use (for the Media Router).  This would unblock projects and feature launches that would otherwise be stalled in the short-term.

2. Figure out what functionality *can* be securely provided by a public API, and what security mechanisms we would need to have in place (e.g., UI notifications?) to launch a public API.

I'll send out a meeting invite shortly.

Thanks,

Yuri

[Yuri Wiitala]

Nathan and Mustafa: I see you're in MTV-43, but I couldn't find any meeting rooms (well, except for a 16-person conference room in MTV-44).  Could you handle arrangements on your end to find a good VC spot?  Thanks!  :)