Capturing compositor output in the browser process

[Marshall Greenblatt]

Hi All,

I'm trying to identify the best approach for creating a WebContents with the following requirements:

A. Render the final composited output to a surface (SkCanvas or GL/D3D texture) that is created and owned by the browser process. For example, an external Content API consumer provides a GL texture ID that receives the output in the browser process.

B. Capture the composited output without first rendering to a native platform window (HWND, X11 Window or NSView). Or, if that's not possible, the native platform window needs to be non-visible (off-screen or otherwise hidden).

C. Works on Windows, Linux and OS X desktop platforms.

I've previously implemented this functionality using the legacy software rendering path (via a custom WebContentsViewPort and BackingStore), but of course that API is gone now.

Based on the current TOT these are the approaches that I've been able to identify so far:

1. Use RenderWidgetHostViewFrameSubscriber to get a media::VideoFrame. This works with the existing RenderWidgetHostView* implementations. However, at least on OS X it doesn't seem that a RenderWidgetHostViewMac can be created without a backing NSView. Will it still work if the NSView (or HWND or X11 Window) is non-visible?

2. Force RenderWidget::CreateOutputSurface to create a CompositorSoftwareOutputDevice and provide a custom RenderWidgetHostView*::OnSwapCompositorFrame implementation. There's no existing API for this currently so taking this approach would be something of a hack. I would probably need to modify CreateWebContentsView and return my own custom WebContentsView implementation similar to the previous legacy approach.

3. Provide my own implementation at the ozone layer. However, this requires Aura which I don't think is currently supported on OS X.

Are there any other potential approaches that I'm missing? What would you recommend?

Thanks,

Marshall

[Dana Jansens]

Android browser compositor does this with an offscreen compositor instance, but I'm not sure exactly where the code is. Maybe one of these fine folks can point it out.

[Alexandre Elias]

Three options come to mind.

1. As Dana alluded to, we did consider using an offscreen compositor instance in the browser process on Android although we didn't end up going that route.  What that would've looked like is to create a new OutputSurface implementation that resembles MailboxOutputSurface (creates and holds a GL texture outside of CC), with an additional method to perform a GL readback.  That would meet your stated requirements.

2. What we ended up going with was the implementation in RenderWidgetHostViewAndroid::CopyFromCompositingSurface which attaches a second DelegatedRendererLayer to the same compositor.  In our case we render to the screen and also want to occasionally get a side output in software bitmap form for our tab switcher thumbnails or Chromecast, so that was the path of least resistance.  In your case, if you take this approach and you actually have no normal display (I'm not clear whether you do or not), you may also need to create a new OutputSurface implementation that just sends everything to /dev/null.

3. A third option maybe worth mentioning is the "Resource-less software rendering" used in Android WebView and triggered from the SynchronousCompositorOutputSurface.  This draws straight from the SkPicture pile held by the renderer impl thread and is the closest supported analogue to legacy software mode.  In your case you'd have to write additional code to ship the resulting pixels to the browser process (WebView doesn't have this problem since it's single-process).  See https://docs.google.com/a/chromium.org/document/d/1jw9Xyuovw32NR73u6uQEVk7-fxNtpS7QWAoDMJhF5W8/edit?pli=1 for the problem space this solves; I imagine it's somewhat analogous to the problems faced by CEF.

Note also that although OS X won't get Aura, it is planned to switch to delegated rendering soonish (in a few months?).

[SIVA KUMAR GUNTURI]

I am trying to support a similar kind of capture mechanism when the views are hidden , using the  RenderWidgetHostViewAndroid::CopyFromcompositingSurface.

Take a look at this bug

https://code.google.com/p/chromium/issues/detail?id=376769.

and the doc https://docs.google.com/document/d/1gRndVmVn7gWJ-rbIHaaOMNsCjSIBn4CAJbZuwLM2ROE/edit?usp=sharing

[Marshall Greenblatt]

Thanks for the feedback everyone. It seems like the best cross-platform approach at this time is to hook into RWHV::OnSwapCompositorFrame and use RWHV::CopyFromCompositingSurface. RenderViewDevToolsAgentHost::OnSwapCompositorFrame seems an interesting model for this (inspecting the ViewHostMsg_SwapCompositorFrame message).  I'll likely have to tweak things a bit so the RHWV thinks it's visible even when the native window is actually hidden.

Are there any upcoming changes planned for OnSwapCompositorFrame or CopyFromCompositingSurface as part of the delegated renderer (ubercompositor) porting effort or OOP-iframe support?

Thanks,

Marshall

[Dana Jansens]

On Thu, Jun 19, 2014 at 10:31 AM, Marshall Greenblatt <magree...@gmail.com> wrote:

Thanks for the feedback everyone. It seems like the best cross-platform approach at this time is to hook into RWHV::OnSwapCompositorFrame and use RWHV::CopyFromCompositingSurface. RenderViewDevToolsAgentHost::OnSwapCompositorFrame seems an interesting model for this (inspecting the ViewHostMsg_SwapCompositorFrame message).  I'll likely have to tweak things a bit so the RHWV thinks it's visible even when the native window is actually hidden.

Are there any upcoming changes planned for OnSwapCompositorFrame or CopyFromCompositingSurface as part of the delegated renderer (ubercompositor) porting effort or OOP-iframe support?

We're been/are working on consolidating our swap and readback paths into all looking like these two, so this sounds like a good way to go.

[Daniel Sievers]

On Thu, Jun 19, 2014 at 10:31 AM, Marshall Greenblatt <magree...@gmail.com> wrote:

Thanks for the feedback everyone. It seems like the best cross-platform approach at this time is to hook into RWHV::OnSwapCompositorFrame and use RWHV::CopyFromCompositingSurface. RenderViewDevToolsAgentHost::OnSwapCompositorFrame seems an interesting model for this (inspecting the ViewHostMsg_SwapCompositorFrame message).  I'll likely have to tweak things a bit so the RHWV thinks it's visible even when the native window is actually hidden.

There is no native window for the RWHV. Hiding or showing a RenderWidget allows for freeing or allocating the resources on the render-compositor side, as well as the 'current frame' which is the delegated frame data held on to by RWHV.

 

Are there any upcoming changes planned for OnSwapCompositorFrame or CopyFromCompositingSurface as part of the delegated renderer (ubercompositor) porting effort or OOP-iframe support?

Thanks,

Marshall

On Thu, Jun 19, 2014 at 7:36 AM, SIVA KUMAR GUNTURI <sivag...@chromium.org> wrote:

I am trying to support a similar kind of capture mechanism when the views are hidden , using the  RenderWidgetHostViewAndroid::CopyFromcompositingSurface.

Take a look at this bug

https://code.google.com/p/chromium/issues/detail?id=376769.

I still don't get the necessity for making live a view while it's hidden, it seems like a contradiction :)

[Marshall Greenblatt]

On Jun 19, 2014, at 7:15 PM, Daniel Sievers <sie...@google.com> wrote:

On Thu, Jun 19, 2014 at 10:31 AM, Marshall Greenblatt <magree...@gmail.com> wrote:

Thanks for the feedback everyone. It seems like the best cross-platform approach at this time is to hook into RWHV::OnSwapCompositorFrame and use RWHV::CopyFromCompositingSurface. RenderViewDevToolsAgentHost::OnSwapCompositorFrame seems an interesting model for this (inspecting the ViewHostMsg_SwapCompositorFrame message).  I'll likely have to tweak things a bit so the RHWV thinks it's visible even when the native window is actually hidden.

There is no native window for the RWHV. Hiding or showing a RenderWidget allows for freeing or allocating the resources on the render-compositor side, as well as the 'current frame' which is the delegated frame data held on to by RWHV.

Can you expand on this more? My understanding is that RWH[V] and WebContents are being merged over time. If that's the case, how would I create a WebContents on OS X using RWHVMac (for example) that doesn't have a backing NSView? Or is there a way to utilize a RenderWidget without a WebContents (and/or with a custom RWHV)?

[SIVA KUMAR GUNTURI]

@Daniel, Sorry I missed to conclude over this.

Below are the different things the copyFromCompisitingSurface will support.
- allow to invoke a read-back while a view is visible only.
- allow to hide a view while a read-back is pending.
- allow a read-back request to wait for a first frame if it was invoked while no frame was received yet.
- allow to lock the front buffer (current frame) when hiding a view (Android has that already).

One more case we wanted to support capture is where we don't have a frame yet but the view is being hidden.
This can be accomplished by talking to the frame evictor, it can decide to defer eviction and trigger a read-back instead.

Due to the limitations of making the view alive while it is hidden to facilitate capture,
we are not considering this approach any more.The same is updated in the bug.

[Daniel Sievers]

On Thu, Jun 19, 2014 at 11:21 PM, SIVA KUMAR GUNTURI <sivag...@chromium.org> wrote:

@Daniel, Sorry I missed to conclude over this.

Below are the different things the copyFromCompisitingSurface will support.
- allow to invoke a read-back while a view is visible only.
- allow to hide a view while a read-back is pending.
- allow a read-back request to wait for a first frame if it was invoked while no frame was received yet.
- allow to lock the front buffer (current frame) when hiding a view (Android has that already).

One more case we wanted to support capture is where we don't have a frame yet but the view is being hidden.
This can be accomplished by talking to the frame evictor, it can decide to defer eviction and trigger a read-back instead.

That functionality sounds great, thanks!

[Dongseong Hwang]

Thank you for good question, Marshall. This question is very important for all embedders.

On Thursday, June 19, 2014 8:48:58 PM UTC+3, Dana Jansens wrote:

On Thu, Jun 19, 2014 at 10:31 AM, Marshall Greenblatt <magree...@gmail.com> wrote:

Thanks for the feedback everyone. It seems like the best cross-platform approach at this time is to hook into RWHV::OnSwapCompositorFrame and use RWHV::CopyFromCompositingSurface. RenderViewDevToolsAgentHost::OnSwapCompositorFrame seems an interesting model for this (inspecting the ViewHostMsg_SwapCompositorFrame message).  I'll likely have to tweak things a bit so the RHWV thinks it's visible even when the native window is actually hidden.

Are there any upcoming changes planned for OnSwapCompositorFrame or CopyFromCompositingSurface as part of the delegated renderer (ubercompositor) porting effort or OOP-iframe support?

We're been/are working on consolidating our swap and readback paths into all looking like these two, so this sounds like a good way to go.

 

I agree on your conclusion, but I have ambiguous points still.

First of all, what is output we want from RWHV::CopyFromCompositingSurface? RWHV::CopyFromCompositingSurface can offer two kinds of services; 1) offer SkBitmap via copying gpu to cpu 2) offer mailbox that refer to texture id in GPU PROCESS

Embedder can not do anything using the mailbox, which is just char[64], because embedder need to use its own gl context outside GPU process as well as chromium code.

However, SkBitmap hit performance prohibitively. Accelerated embedder finally upload the SkBitmap to gpu again. Nobody wants to copy gpu->cpu->gpu.

Here is my imagination.

1. Embedder per platform passes GLSurfaceHandle to browser process which wraps native pbuffer. see native_widget_type.h Linux and cros uses xpixmap, window uses HWND, ios uses IOSurface, Android uses Android_native_window, ozone uses just uint64

2. RWHV::CopyFromCompositingSurface copies the frame texture to the GLSurfaceHandle in GPU process

3. Embedder use the GLSurfaceHandle outside chromium code in the process same to browser process.

The main issue is how to synchronize between #2 and #3. Chromium can not know when Embedder reads the GLSurfaceHandle. Do we need double buffering or triple buffering like Android SurfaceFlinger? The relation is almost same; embedder : gpu process = SurfaceFlinger : Suface in any activity.

WDYT, guys?

[Alexey Baskakov]

Hi, everybody!

> what is output we want from RWHV::CopyFromCompositingSurface?

I would add the 3rd case: sometimes it's nice to have scoped_refptr<cc::Layer> readback_layer as an output.
To add some background filters and use it in Browser-side UI, for example.

Loyso.

[Dana Jansens]

I feel like I'm missing something, as this just skyrocketed in complexity. Is a flush() in the GPU process before returning the texture to the content embedder not enough to ensure it is complete? Chromium doesn't need to know when the embedder reads the texture, as it's not using the texture itself, just drawing into it and handing it back, right?

[Dana Jansens]

On Tue, Jul 1, 2014 at 11:03 AM, Alexey Baskakov <loy...@gmail.com> wrote:

Hi, everybody!

> what is output we want from RWHV::CopyFromCompositingSurface?

I would add the 3rd case: sometimes it's nice to have scoped_refptr<cc::Layer> readback_layer as an output.
To add some background filters and use it in Browser-side UI, for example.

How can a cc::Layer the output of a readback, as it does not have any pixel content on its own, and is just a bunch of flags for how to present/clip/transform/etc content. At best, you'd have a TextureLayer with a texture mailbox set on the Layer. Is this to avoid putting the texture into a cc::TextureLayer yourself? I'd suggest just doing that.

[Dongseong Hwang]

You're right. flush() in the GPU process is enough and Chromium doesn't need to know when the embedder reads the texture. But embedder has still issue. Embedder must ensure to read the texture on browser process between after-copying and before-copying-next-frame on gpu process.

one of naive idea is ;

1. embedder create 2~4 mailboxes like webgl. the backend of mailboxes is platform surface. embedder manages the mailbox ring-buffer like webgl.

2. embedder issues the mailboxes as CopyResult to compositor in turn.

3. when returning CopyResult, embedder use it.

I guess some change in compositor and gpu process are required to support platform surface mailbox.

- DS

[Dana Jansens]

My impression of use cases for this was things that would not be reading a constant stream of textures. If they want to give those textures back to chromium to copy into again, can it again flush after not using the texture again? Nothing here is as performant as sync points, I guess we'd need a compelling use case to add some external sync-point-like thing.

 

- DS

[Dongseong Hwang]

On Wednesday, July 2, 2014 11:29:19 PM UTC+3, Dana Jansens wrote:

My impression of use cases for this was things that would not be reading a constant stream of textures. If they want to give those textures back to chromium to copy into again, can it again flush after not using the texture again? Nothing here is as performant as sync points, I guess we'd need a compelling use case to add some external sync-point-like thing.

 

Holy-grail use case all thirdparty embedder want is to make WebView integrated with toolkit such as Qt, GTK, Cocoa, EFL, etc. And embedders want WebView as fast as Chrome browser. So if Chromium renders some contents 60FPS, embedder want to read the copied texture 60 times a sec with minimal overhead.

In addition, chromium never have had such use case until now; sharing hardware surface inter-process. Briefly what embedder want is that gpu process draws results on hardware surface and browser process uses it in an alien GL context.

Yep, embedder might need some external sync-point-like thing.

[Dana Jansens]

This sounds like we're getting into the territory of how content owners want to support chromium being embedded, which is probably something better discussed on chromium-dev where they all read.

[Marshall Greenblatt]

+chromium-dev (cc)

-graphics-dev (bcc)

To better understand the use case for browser process texture output you can read this article:

http://coherent-labs.com/what-developers-should-consider-when-using-chromium-embedded-framework-cef-in-their-games/

I'm not associated with that company, but it seems they're using some type of WebKit hybrid because Chromium doesn't currently offer the necessary performance and functionality.

[Alexey Baskakov]

I meant any suitable Layer-derived class, of couse.

Now I see that It would be easy to create TextureLayer once you have TextureMailBox for readback data.
So, my 3rd case is redundant. Thanks for pointing that out!

Also I meant the case where RenderWidgetHostViewAndroid::CopyFromCompositingSurface uses DelegatedRendererLayer effectively. What's the difference to hold that layer instead of TextureLayer?

Thanks, Loyso.
Yandex Browser Team.

[Dana Jansens]

On Thu, Jul 3, 2014 at 3:24 AM, Alexey Baskakov <loy...@gmail.com> wrote:

I meant any suitable Layer-derived class, of couse.

Now I see that It would be easy to create TextureLayer once you have TextureMailBox for readback data.
So, my 3rd case is redundant. Thanks for pointing that out!

Also I meant the case where RenderWidgetHostViewAndroid::CopyFromCompositingSurface uses DelegatedRendererLayer effectively. What's the difference to hold that layer instead of TextureLayer?

The output of a copy request will be a texture in either case.  The type of layer internally has no effect on what the copy request produces, so you'd put the result in a TextureLayer if you wanted to show it in the UI.