Screenshots for Headless Chrome

[Eric Seckler]

Hi all,

Following up on your feedback regarding the viewport overrides for partial screenshotting, we've been rethinking our requirements and came up with an alternative approach that doesn't require overrides.

In particular, we now believe that it is important that the page does not observe that we take only a partial screenshot. Using the visual viewport, as in the previous proposal, does not easily allow us to do this. Instead, we are now looking to resize the CompositorFrame sent from the renderer in combination with a custom transform (scale+offset) applied to the CompositorFrame's content in the renderer.

Feel free to have a look at a revised design doc and let us know whether you think this sounds feasible.

Thanks!

Eric

[dan...@chromium.org]

+boliu for thoughts about maybe merging this with resourceless software draws?

[Bo Liu]

The part about hiding the viewport change to the page does sound like webview's resourceless software draw. Doc mentions external_transform_, but not external_viewport_. external_viewport_ basically allows setting a custom viewport at draw time as well.

I want to call out some limitations with webview's approach

* record_whole_document can be really really slow, I think last time I checked, a huge text page like the complete buildbot stdio can take half a minute to record on a two year old phone

* fixed position, and pseudo fixed position elements by js might give unexpected results with external_viewport

* I can imagine some web features like responsive images might not actually be responsive since external_viewport_ cannot affect page layout (just an example, I have no idea how responsive image is actually implemened)

Probably don't need to copy "resourceless" behavior though. Just do a normal software draw.

[Eric Seckler]

Thanks Bo! Sounds like we should try to avoid record_whole_document. As far as we know, WebView's onDraw approach also doesn't support WebGL, videos, and similar. (Plus, we don't really want to limit us to software-only draws in the future.) This, added to the limitations you mention, makes us believe that it's not the right approach for us.

Regarding external_viewport_, this would also only work in connection with software draw, IIRC?

[Bo Liu]

That's what I meant by that last line. Ignore both the "resourceless" and the "software" parts. Webview does hardware onDraw with tiles (ie resources) as well. All the limitations I listed apply regardless of hardware or software draw though.

On Mon, Aug 1, 2016 at 11:03 AM, Eric Seckler <esec...@chromium.org> wrote:

Thanks Bo! Sounds like we should try to avoid record_whole_document. As far as we know, WebView's onDraw approach also doesn't support WebGL, videos, and similar. (Plus, we don't really want to limit us to software-only draws in the future.) This, added to the limitations you mention, makes us believe that it's not the right approach for us.

Regarding external_viewport_, this would also only work in connection with software draw, IIRC?

It works fine for hardware draw in theory. But in practice the tiling system assumes viewport only changes incrementally, ie in response to user input. It doesn't work so well if viewport "teleports" to a different part of the page, then there may not be tiles ready, at the right scale, etc. But if you are willing to wait for raster to finish first before draw (or maybe headless don't need tiles at all?), it should be fine.

[Eric Seckler]

On Mon, Aug 1, 2016 at 7:45 PM Bo Liu <bo...@chromium.org> wrote:

That's what I meant by that last line. Ignore both the "resourceless" and the "software" parts. Webview does hardware onDraw with tiles (ie resources) as well. All the limitations I listed apply regardless of hardware or software draw though.

Ah, thanks, we didn't realize that. Does it also support OOPIFs? We were assuming this wasn't supported, since it draws in a renderer process. Headless uses a separate browser process, and we'd like to support OOPIF (e.g. to use headless for test automation under close-to-real environments).

 

It works fine for hardware draw in theory. But in practice the tiling system assumes viewport only changes incrementally, ie in response to user input. It doesn't work so well if viewport "teleports" to a different part of the page, then there may not be tiles ready, at the right scale, etc. But if you are willing to wait for raster to finish first before draw (or maybe headless don't need tiles at all?), it should be fine.

For headless, we're happy with tiles. Eventually, we would like to disable rasterization outside the screenshot area, and using tiles with an interest area padding of 0px seems like a good solution for that. We're also planning on limiting raster/paint to occur only when we take a screenshot.

On Mon, Aug 1, 2016 at 11:03 AM, Eric Seckler <esec...@chromium.org> wrote:

Sounds like we should try to avoid record_whole_document. 

We've been looking into this further. When record_whole_document is disabled, the recording interest area seems to be calculated using FrameView::visibleContentRect() plus a hardcoded 4000px padding. Similarly to the rasterization, we'd like to set this interest area to the exact Rect needed for the screenshot (could be larger or smaller than the FrameView's visibleContentRect). Since we don't want to change the FrameView size to the screenshot area (this would be observable by the page AFAIK), we can't think of a better way than adding some form of override for this within CompositedLayerMapping (or an alternative mode to calculate it, taking into account scale/bounds of the screenshot area). WDYT?

[Bo Liu]

On Tue, Aug 2, 2016 at 6:26 AM, Eric Seckler <esec...@chromium.org> wrote:

On Mon, Aug 1, 2016 at 7:45 PM Bo Liu <bo...@chromium.org> wrote:

That's what I meant by that last line. Ignore both the "resourceless" and the "software" parts. Webview does hardware onDraw with tiles (ie resources) as well. All the limitations I listed apply regardless of hardware or software draw though.

Ah, thanks, we didn't realize that. Does it also support OOPIFs? We were assuming this wasn't supported, since it draws in a renderer process. Headless uses a separate browser process, and we'd like to support OOPIF (e.g. to use headless for test automation under close-to-real environments).

OOPIF in webview will need browser side changes. I don't see why it can't work though.

 

 

It works fine for hardware draw in theory. But in practice the tiling system assumes viewport only changes incrementally, ie in response to user input. It doesn't work so well if viewport "teleports" to a different part of the page, then there may not be tiles ready, at the right scale, etc. But if you are willing to wait for raster to finish first before draw (or maybe headless don't need tiles at all?), it should be fine.

For headless, we're happy with tiles. Eventually, we would like to disable rasterization outside the screenshot area, and using tiles with an interest area padding of 0px seems like a good solution for that. We're also planning on limiting raster/paint to occur only when we take a screenshot.

On Mon, Aug 1, 2016 at 11:03 AM, Eric Seckler <esec...@chromium.org> wrote:

Sounds like we should try to avoid record_whole_document. 

We've been looking into this further. When record_whole_document is disabled, the recording interest area seems to be calculated using FrameView::visibleContentRect() plus a hardcoded 4000px padding. Similarly to the rasterization, we'd like to set this interest area to the exact Rect needed for the screenshot (could be larger or smaller than the FrameView's visibleContentRect). Since we don't want to change the FrameView size to the screenshot area (this would be observable by the page AFAIK), we can't think of a better way than adding some form of override for this within CompositedLayerMapping (or an alternative mode to calculate it, taking into account scale/bounds of the screenshot area). WDYT?

I think recording is done in vector form (?), so there isn't really a scale, which means recording a long text page is going to be slow no matter.

[Sami Kyostila]

ti 2. elokuuta 2016 klo 18.15 Bo Liu <bo...@chromium.org> kirjoitti:

On Tue, Aug 2, 2016 at 6:26 AM, Eric Seckler <esec...@chromium.org> wrote:

On Mon, Aug 1, 2016 at 7:45 PM Bo Liu <bo...@chromium.org> wrote:

That's what I meant by that last line. Ignore both the "resourceless" and the "software" parts. Webview does hardware onDraw with tiles (ie resources) as well. All the limitations I listed apply regardless of hardware or software draw though.

Ah, thanks, we didn't realize that. Does it also support OOPIFs? We were assuming this wasn't supported, since it draws in a renderer process. Headless uses a separate browser process, and we'd like to support OOPIF (e.g. to use headless for test automation under close-to-real environments).

OOPIF in webview will need browser side changes. I don't see why it can't work though.

Is there a benefit to using OnDraw in the browser process as opposed to normal CopyOutputRequest path? We're fine with the readback being asynchronous, but maybe there are other advantages?

 

 

 

It works fine for hardware draw in theory. But in practice the tiling system assumes viewport only changes incrementally, ie in response to user input. It doesn't work so well if viewport "teleports" to a different part of the page, then there may not be tiles ready, at the right scale, etc. But if you are willing to wait for raster to finish first before draw (or maybe headless don't need tiles at all?), it should be fine.

For headless, we're happy with tiles. Eventually, we would like to disable rasterization outside the screenshot area, and using tiles with an interest area padding of 0px seems like a good solution for that. We're also planning on limiting raster/paint to occur only when we take a screenshot.

On Mon, Aug 1, 2016 at 11:03 AM, Eric Seckler <esec...@chromium.org> wrote:

Sounds like we should try to avoid record_whole_document. 

We've been looking into this further. When record_whole_document is disabled, the recording interest area seems to be calculated using FrameView::visibleContentRect() plus a hardcoded 4000px padding. Similarly to the rasterization, we'd like to set this interest area to the exact Rect needed for the screenshot (could be larger or smaller than the FrameView's visibleContentRect). Since we don't want to change the FrameView size to the screenshot area (this would be observable by the page AFAIK), we can't think of a better way than adding some form of override for this within CompositedLayerMapping (or an alternative mode to calculate it, taking into account scale/bounds of the screenshot area). WDYT?

I think recording is done in vector form (?), so there isn't really a scale, which means recording a long text page is going to be slow no matter.

I think Eric meant we'd try to ~only record the area that is being screenshotted. If someone wants to screenshot the whole document, then that's going to be slow but there's really way around that.

In any case, I think we'd need some more control over the computed interest area to be able to record something larger than visibleContentRect() but smaller than the entire document.

- Sami

[Eric Seckler]

On Tue, Aug 2, 2016 at 6:31 PM Sami Kyostila <skyo...@chromium.org> wrote:

I think recording is done in vector form (?), so there isn't really a scale, which means recording a long text page is going to be slow no matter.

I think Eric meant we'd try to ~only record the area that is being screenshotted. If someone wants to screenshot the whole document, then that's going to be slow but there's really [no] way around that.  

 

In any case, I think we'd need some more control over the computed interest area to be able to record something larger than visibleContentRect() but smaller than the entire document.

Correct. The reason I mentioned the scale as well is that the region that needs to be recorded (in CSS px) shrinks with larger scale and expands with smaller scale. Thus, we probably need to be aware of the screenshot scale for computing the recording interest area.

[Bo Liu]

On Tue, Aug 2, 2016 at 10:30 AM, Sami Kyostila <skyo...@chromium.org> wrote:

ti 2. elokuuta 2016 klo 18.15 Bo Liu <bo...@chromium.org> kirjoitti:

On Tue, Aug 2, 2016 at 6:26 AM, Eric Seckler <esec...@chromium.org> wrote:

On Mon, Aug 1, 2016 at 7:45 PM Bo Liu <bo...@chromium.org> wrote:

That's what I meant by that last line. Ignore both the "resourceless" and the "software" parts. Webview does hardware onDraw with tiles (ie resources) as well. All the limitations I listed apply regardless of hardware or software draw though.

Ah, thanks, we didn't realize that. Does it also support OOPIFs? We were assuming this wasn't supported, since it draws in a renderer process. Headless uses a separate browser process, and we'd like to support OOPIF (e.g. to use headless for test automation under close-to-real environments).

OOPIF in webview will need browser side changes. I don't see why it can't work though.

Is there a benefit to using OnDraw in the browser process as opposed to normal CopyOutputRequest path? We're fine with the readback being asynchronous, but maybe there are other advantages?

Not really.

external_transform_ and external_viewport_ are tied to OnDraw right now, but they don't have to be. And sounds like if you are building controls for which rect to record and raster, those are not really needed anyway.