Separate layers for animated gifs?

[Christopher Cameron]

Videos get their own separate layers. From what I can tell, animated gifs never do.

Has there been any investigation into creating separate layers for animated gifs? A quick search in crbug didn't reveal anything obvious in that area. Most work I could find has been on issues with caching (e.g, caching all of the frames can hurt).

It seems that (at least on macOS) doing so would decrease our damage rects (which is always good for power savings).

[Justin Novosad]

I think exploring this is a really good idea. I wonder whether it would also be reasonable to have a compositing layer that is backed by a GpuMemoryBuffer (or several GpuMemoryBuffers for swapchaining), and have the GIF decoder write directly into it, and possibly use hardware overlay. 

[Jayson Adams]

I'm working on reducing Chrome's power usage. I'm available to work on this if there's no one with spare cycles.

[Chris Harrelson]

+Khushal, who worked in images-on-compositor thread in the past. +pdr as TL for paint/compositing.

I don't think this has been tried.

Do you have data on savings, and how common this is across the web? Also, allocating a new composited layer is by no means free, there is plenty of overhead for it. I wonder if instead we can just optimize the code for damage within an existing layer.

[Dale Curtis]

cblume@ wrote several related docs on the state of images which might be relevant:

Smoother, More Efficient Animation Frame Caching

Animated Images & Surpassing the Cache Budget
Allow Gif Time Dilation (this one is probably most relevant. Ignore the "gif" part.)

Unifying the video and animated image paths would be quite nice if possible. If we could use this for HDR images too would let us use OS overlays for tonemapping too.

 -dale

[Jayson Adams]

On Fri, Aug 27, 2021 at 10:50 AM Chris Harrelson <chri...@chromium.org> wrote:

+Khushal, who worked in images-on-compositor thread in the past. +pdr as TL for paint/compositing.

I don't think this has been tried.

Do you have data on savings, and how common this is across the web? Also, allocating a new composited layer is by no means free, there is plenty of overhead for it. I wonder if instead we can just optimize the code for damage within an existing layer.

I'm not sure what you mean when you ask how common this is. Are you asking about the prevalence of animated gifs?

If you run Chromium with 
--show-surface-damage-rects and go to this page you'll see the damage rect encloses the animated gif:

On a Mac, running with QuartzDebug shows a very large update region getting pushed to the screen (see the attached QuartzDebugScreenUpdates.mov). We're consuming a lot more power than we need to as a result. The issue is magnified if the page contains two or more animated gifs.

I found another instance of what I thought was this same problem but I don't think there are any animated gifs involved(?). If you go to https://www.bookbaby.com/myaccount/login.aspx you'll see the damage rect encloses the insertion point. However in the attached screen recording the insertion point flashing is causing a similarly large screen update as in the animated gif case. Is this a separate problem?

Best,

__jayson

[Chris Harrelson]

On Fri, Aug 27, 2021 at 11:23 AM Jayson Adams <shr...@chromium.org> wrote:

On Fri, Aug 27, 2021 at 10:50 AM Chris Harrelson <chri...@chromium.org> wrote:

+Khushal, who worked in images-on-compositor thread in the past. +pdr as TL for paint/compositing.

I don't think this has been tried.

Do you have data on savings, and how common this is across the web? Also, allocating a new composited layer is by no means free, there is plenty of overhead for it. I wonder if instead we can just optimize the code for damage within an existing layer.

I'm not sure what you mean when you ask how common this is. Are you asking about the prevalence of animated gifs?

Yes, and how much it hurts performance on the pages where it's present.

 

If you run Chromium with 
--show-surface-damage-rects and go to this page you'll see the damage rect encloses the animated gif:

On a Mac, running with QuartzDebug shows a very large update region getting pushed to the screen (see the attached QuartzDebugScreenUpdates.mov). We're consuming a lot more power than we need to as a result. The issue is magnified if the page contains two or more animated gifs.

I found another instance of what I thought was this same problem but I don't think there are any animated gifs involved(?). If you go to https://www.bookbaby.com/myaccount/login.aspx you'll see the damage rect encloses the insertion point. However in the attached screen recording the insertion point flashing is causing a similarly large screen update as in the animated gif case. Is this a separate problem?

I see. I forget the details in Viz, but isn't there a way to contain the damage rect to a smaller regions that just contains the animated GIF?

[Maggie Chen]

In Viz surface aggregation, damages from all surfaces are merged into one "root damage rect ". This is the final rect for the Renderer to update and display. Command line switch --show-aggregated-damage will shows the damage rect after surface aggregation. If we run this page on Windows, the only part that is updated should be the gif rect.

I know this Viz root damage rect is not used on Mac, I just tried that gif again on Mac with command line --show-mac-overlay-borders. Below is what it shows. It updates more than the root damage rect.

--
To unsubscribe from this group and stop receiving emails from it, send an email to graphics-dev...@chromium.org.

[Jayson Adams]

I originally thought this might be a Mac problem but then I ran Chromium with 
--show-composited-layer-borders, which I assume is cross-platform. I thought I was getting the same rects as the Mac overlay borders but on closer inspection they don't match up.

[Maggie Chen]

I think it's because my browser windows size is different. So the Mac overlay borders look different from Jayson's screen shot. If I compare the borders in --show-composited-layer-borders and in --show-mac-overlay-borders, they match each other on my Mac.

[Chris Harrelson]

On Fri, Aug 27, 2021 at 5:04 PM Maggie Chen <mag...@google.com> wrote:

I think it's because my browser windows size is different. So the Mac overlay borders look different from Jayson's screen shot. If I compare the borders in --show-composited-layer-borders and in --show-mac-overlay-borders, they match each other on my Mac.

On Fri, Aug 27, 2021 at 4:08 PM Jayson Adams <shr...@chromium.org> wrote:

I originally thought this might be a Mac problem but then I ran Chromium with 
--show-composited-layer-borders, which I assume is cross-platform. I thought I was getting the same rects as the Mac overlay borders but on closer inspection they don't match up.

--show-composited-layer-borders is cross-platform. The only compositing differences between Mac and non-Mac is in the aggregate step of the pipeline.

 

On Fri, Aug 27, 2021 at 3:44 PM Maggie Chen <mag...@google.com> wrote:

In Viz surface aggregation, damages from all surfaces are merged into one "root damage rect ". This is the final rect for the Renderer to update and display. Command line switch --show-aggregated-damage will shows the damage rect after surface aggregation. If we run this page on Windows, the only part that is updated should be the gif rect.

I know this Viz root damage rect is not used on Mac, I just tried that gif again on Mac with command line --show-mac-overlay-borders. Below is what it shows. It updates more than the root damage rect.

Exactly - this seems like something we can try to improve on Mac. (I know it's tricky due to the use of Core Animations, but still. :) )

[Ken Russell]

On Fri, Aug 27, 2021 at 5:45 PM Chris Harrelson <chri...@chromium.org> wrote:

On Fri, Aug 27, 2021 at 5:04 PM Maggie Chen <mag...@google.com> wrote:

I think it's because my browser windows size is different. So the Mac overlay borders look different from Jayson's screen shot. If I compare the borders in --show-composited-layer-borders and in --show-mac-overlay-borders, they match each other on my Mac.

On Fri, Aug 27, 2021 at 4:08 PM Jayson Adams <shr...@chromium.org> wrote:

I originally thought this might be a Mac problem but then I ran Chromium with 
--show-composited-layer-borders, which I assume is cross-platform. I thought I was getting the same rects as the Mac overlay borders but on closer inspection they don't match up.

--show-composited-layer-borders is cross-platform. The only compositing differences between Mac and non-Mac is in the aggregate step of the pipeline.

 

On Fri, Aug 27, 2021 at 3:44 PM Maggie Chen <mag...@google.com> wrote:

In Viz surface aggregation, damages from all surfaces are merged into one "root damage rect ". This is the final rect for the Renderer to update and display. Command line switch --show-aggregated-damage will shows the damage rect after surface aggregation. If we run this page on Windows, the only part that is updated should be the gif rect.

I know this Viz root damage rect is not used on Mac, I just tried that gif again on Mac with command line --show-mac-overlay-borders. Below is what it shows. It updates more than the root damage rect.

Exactly - this seems like something we can try to improve on Mac. (I know it's tricky due to the use of Core Animations, but still. :) )

Would partial swap need to be supported on Core Animation layers in order to support this?

[Christopher Cameron]

My suspicion is that the biggest gain will be from the more precise damage during swap. The area of pixels one changes on the screen has a substantial impact on power on macOS. I suspect this would have a more limited impact on other platforms (e.g, I hear that on Windows, moving non-overlay videos to their own layers didn't produce much gain).

Is there a quick-and-hacky way to force animated gifs to have their own layers? With that we could just measure if this makes a difference and then further invest or discard it accordingly. (In fact, just making layers for <img> elements of a size greater than some constant would be enough to get power numbers).

RE partial swap: Mac doesn't have partial swap -- individual CALayers can only take full damage. (And I think this is as it should be -- if we just want to damage the rectangle on the screen for an image, and we requested a feature to do this, the response would rightly be "why aren't you making that image its own CALayer?").

[Philip Rogers]

For testing, add the following style to the <img> tag to cause it to get a cc::Layer:

  style="will-change: transform;"

It is relatively easy (<1 week) to automatically promote large animated images with CompositeAfterPaint (launching M94). A potential worry is the memory cost (see: gpu memory ablation study). Want to collaborate on a finch trial in Q4?

[Khushal Sagar]

On Sat, Aug 28, 2021 at 12:15 PM Philip Rogers <p...@chromium.org> wrote:

For testing, add the following style to the <img> tag to cause it to get a cc::Layer:

  style="will-change: transform;"

It is relatively easy (<1 week) to automatically promote large animated images with CompositeAfterPaint (launching M94). A potential worry is the memory cost (see: gpu memory ablation study). Want to collaborate on a finch trial in Q4?

Could you clarify the CAP dependency here? I assumed all we'll need is to make the presence of an animated image a compositing trigger. Add the trigger here and query whether the LayoutObject is an animated image similar to the logic here. It should automatically hit the directly composited image path and associated optimizations as well, correct?

 

On Fri, Aug 27, 2021 at 8:52 PM Christopher Cameron <ccam...@google.com> wrote:

My suspicion is that the biggest gain will be from the more precise damage during swap. The area of pixels one changes on the screen has a substantial impact on power on macOS. I suspect this would have a more limited impact on other platforms (e.g, I hear that on Windows, moving non-overlay videos to their own layers didn't produce much gain).

Is there a quick-and-hacky way to force animated gifs to have their own layers? With that we could just measure if this makes a difference and then further invest or discard it accordingly. (In fact, just making layers for <img> elements of a size greater than some constant would be enough to get power numbers).

RE partial swap: Mac doesn't have partial swap -- individual CALayers can only take full damage. (And I think this is as it should be -- if we just want to damage the rectangle on the screen for an image, and we requested a feature to do this, the response would rightly be "why aren't you making that image its own CALayer?").

Can't we damage a CALayer partially? That would avoid the additional memory cost to layerize an element just to avoid unnecessary damage during swap. Android has a very similar API (setDamageRegion) to specify which part of a layer changed during a frame. This seems generally useful for any compositing system and we could easily use it to leverage partial swap on mac.

[Philip Rogers]

On Mon, Aug 30, 2021 at 9:18 AM Khushal Sagar <khusha...@chromium.org> wrote:

On Sat, Aug 28, 2021 at 12:15 PM Philip Rogers <p...@chromium.org> wrote:

For testing, add the following style to the <img> tag to cause it to get a cc::Layer:

  style="will-change: transform;"

It is relatively easy (<1 week) to automatically promote large animated images with CompositeAfterPaint (launching M94). A potential worry is the memory cost (see: gpu memory ablation study). Want to collaborate on a finch trial in Q4?

Could you clarify the CAP dependency here? I assumed all we'll need is to make the presence of an animated image a compositing trigger. Add the trigger here and query whether the LayoutObject is an animated image similar to the logic here. It should automatically hit the directly composited image path and associated optimizations as well, correct?

Pre-CompositeAfterPaint, compositing changes paint order. This is known as the "fundamental compositing bug" (crbug.com/370604) and it prevents us from compositing arbitrary content. Here is an example that would break if we composited animated images pre-CompositeAfterPaint:

  <!doctype html>
  <img src="https://upload.wikimedia.org/wikipedia/commons/1/1d/Newtons_cradle.gif">
  <div style="width: 100px; height: 100px; background: lightblue; display: inline-block; margin-left: -50px;">This should be on top.</div>

CompositeAfterPaint fixes this issue by running the overlap test after painting which lets us composite the content that overlaps the animating image, preserving paint order.

[Khushal Sagar]

On Mon, Aug 30, 2021 at 12:33 PM Philip Rogers <p...@chromium.org> wrote:

On Mon, Aug 30, 2021 at 9:18 AM Khushal Sagar <khusha...@chromium.org> wrote:

On Sat, Aug 28, 2021 at 12:15 PM Philip Rogers <p...@chromium.org> wrote:

For testing, add the following style to the <img> tag to cause it to get a cc::Layer:

  style="will-change: transform;"

It is relatively easy (<1 week) to automatically promote large animated images with CompositeAfterPaint (launching M94). A potential worry is the memory cost (see: gpu memory ablation study). Want to collaborate on a finch trial in Q4?

Could you clarify the CAP dependency here? I assumed all we'll need is to make the presence of an animated image a compositing trigger. Add the trigger here and query whether the LayoutObject is an animated image similar to the logic here. It should automatically hit the directly composited image path and associated optimizations as well, correct?

Pre-CompositeAfterPaint, compositing changes paint order. This is known as the "fundamental compositing bug" (crbug.com/370604) and it prevents us from compositing arbitrary content. Here is an example that would break if we composited animated images pre-CompositeAfterPaint:

  <!doctype html>
  <img src="https://upload.wikimedia.org/wikipedia/commons/1/1d/Newtons_cradle.gif">
  <div style="width: 100px; height: 100px; background: lightblue; display: inline-block; margin-left: -50px;">This should be on top.</div>

CompositeAfterPaint fixes this issue by running the overlap test after painting which lets us composite the content that overlaps the animating image, preserving paint order.

Thanks for clarifying. I sort of assumed that we can already layerize images because we have a "directly composited image" path. Didn't realize it was limited to images which have their own stacking context. I'm excited to hear CAP is taking away this restriction!

IMO supporting partial swap on mac if we have an API that allows partially damaging layers is a better area to optimize. Would setNeedsDisplayInRect work? It will address use-cases beyond gifs.

But if we can promote GIFs to layers it could help us avoid a few redundant copies for these animations (that have been on the wishlist). It might end up being better for memory in some cases as well :

- Don't raster tiles for these images. We can use the decoded image textures during viz compositing directly, that would benefit platforms beyond mac. Also avoid allocating textures for tiles.

- Decode directly into GMBs to avoid a texture upload. +Chris Blume had started work on this. Should work for Mac/CrOS and maybe also Android.

[Ken Russell]

On Mon, Aug 30, 2021 at 10:11 AM Khushal Sagar <khusha...@chromium.org> wrote:

On Mon, Aug 30, 2021 at 12:33 PM Philip Rogers <p...@chromium.org> wrote:

On Mon, Aug 30, 2021 at 9:18 AM Khushal Sagar <khusha...@chromium.org> wrote:

On Sat, Aug 28, 2021 at 12:15 PM Philip Rogers <p...@chromium.org> wrote:

For testing, add the following style to the <img> tag to cause it to get a cc::Layer:

  style="will-change: transform;"

It is relatively easy (<1 week) to automatically promote large animated images with CompositeAfterPaint (launching M94). A potential worry is the memory cost (see: gpu memory ablation study). Want to collaborate on a finch trial in Q4?

Could you clarify the CAP dependency here? I assumed all we'll need is to make the presence of an animated image a compositing trigger. Add the trigger here and query whether the LayoutObject is an animated image similar to the logic here. It should automatically hit the directly composited image path and associated optimizations as well, correct?

Pre-CompositeAfterPaint, compositing changes paint order. This is known as the "fundamental compositing bug" (crbug.com/370604) and it prevents us from compositing arbitrary content. Here is an example that would break if we composited animated images pre-CompositeAfterPaint:

  <!doctype html>
  <img src="https://upload.wikimedia.org/wikipedia/commons/1/1d/Newtons_cradle.gif">
  <div style="width: 100px; height: 100px; background: lightblue; display: inline-block; margin-left: -50px;">This should be on top.</div>

CompositeAfterPaint fixes this issue by running the overlap test after painting which lets us composite the content that overlaps the animating image, preserving paint order.

Thanks for clarifying. I sort of assumed that we can already layerize images because we have a "directly composited image" path. Didn't realize it was limited to images which have their own stacking context. I'm excited to hear CAP is taking away this restriction!

IMO supporting partial swap on mac if we have an API that allows partially damaging layers is a better area to optimize. Would setNeedsDisplayInRect work? It will address use-cases beyond gifs.

My understanding is that that API doesn't work / help with IOSurface-backed CALayers, which is the common backing type that Chrome uses:

https://source.chromium.org/chromium/chromium/src/+/main:ui/accelerated_widget_mac/ca_renderer_layer_tree.mm;l=922?q=CALayer%20setContents

It probably only helps with layers rendered to by Core Graphics.

The longstanding issue is that CALayer updates have been "all or nothing" - back when Chrome used OpenGL to render its main CALayer, the only option was a full buffer swap for the smallest update. ccameron's work to support multiple CALayers and separate updates was the foundational change that yielded huge power savings, for example with Docs' or Gmail's blinking caret.

Chris (Cameron) might know whether we've tried the experiment to use setNeedsDisplayInRect for rasterization damage.

[Christopher Cameron]

On Mon, Aug 30, 2021 at 10:40 AM Ken Russell <k...@chromium.org> wrote:

On Mon, Aug 30, 2021 at 10:11 AM Khushal Sagar <khusha...@chromium.org> wrote:

On Mon, Aug 30, 2021 at 12:33 PM Philip Rogers <p...@chromium.org> wrote:

On Mon, Aug 30, 2021 at 9:18 AM Khushal Sagar <khusha...@chromium.org> wrote:

On Sat, Aug 28, 2021 at 12:15 PM Philip Rogers <p...@chromium.org> wrote:

For testing, add the following style to the <img> tag to cause it to get a cc::Layer:

  style="will-change: transform;"

It is relatively easy (<1 week) to automatically promote large animated images with CompositeAfterPaint (launching M94). A potential worry is the memory cost (see: gpu memory ablation study). Want to collaborate on a finch trial in Q4?

Could you clarify the CAP dependency here? I assumed all we'll need is to make the presence of an animated image a compositing trigger. Add the trigger here and query whether the LayoutObject is an animated image similar to the logic here. It should automatically hit the directly composited image path and associated optimizations as well, correct?

Pre-CompositeAfterPaint, compositing changes paint order. This is known as the "fundamental compositing bug" (crbug.com/370604) and it prevents us from compositing arbitrary content. Here is an example that would break if we composited animated images pre-CompositeAfterPaint:

  <!doctype html>
  <img src="https://upload.wikimedia.org/wikipedia/commons/1/1d/Newtons_cradle.gif">
  <div style="width: 100px; height: 100px; background: lightblue; display: inline-block; margin-left: -50px;">This should be on top.</div>

CompositeAfterPaint fixes this issue by running the overlap test after painting which lets us composite the content that overlaps the animating image, preserving paint order.

Thanks for clarifying. I sort of assumed that we can already layerize images because we have a "directly composited image" path. Didn't realize it was limited to images which have their own stacking context. I'm excited to hear CAP is taking away this restriction!

IMO supporting partial swap on mac if we have an API that allows partially damaging layers is a better area to optimize. Would setNeedsDisplayInRect work? It will address use-cases beyond gifs.

My understanding is that that API doesn't work / help with IOSurface-backed CALayers, which is the common backing type that Chrome uses:

https://source.chromium.org/chromium/chromium/src/+/main:ui/accelerated_widget_mac/ca_renderer_layer_tree.mm;l=922?q=CALayer%20setContents

It probably only helps with layers rendered to by Core Graphics.

The longstanding issue is that CALayer updates have been "all or nothing" - back when Chrome used OpenGL to render its main CALayer, the only option was a full buffer swap for the smallest update. ccameron's work to support multiple CALayers and separate updates was the foundational change that yielded huge power savings, for example with Docs' or Gmail's blinking caret.

Chris (Cameron) might know whether we've tried the experiment to use setNeedsDisplayInRect for rasterization damage.

Indeed setNeedsDisplayInRect "is for a different thing". Before we went to the "everything is a CALayer" approach, there were like 3 or 4 Chrome releases where we hacked in some form of partial swap support for the backbuffer. It was messy (like a stack of layers with progressive amounts of damage).

I don't feel that we're hurting for a lack of CALayer partial damage in the API. When the need for the API arises, like this use case, it's generally a sign that we're holding it wrong (see below).

 

But if we can promote GIFs to layers it could help us avoid a few redundant copies for these animations (that have been on the wishlist). It might end up being better for memory in some cases as well :

- Don't raster tiles for these images. We can use the decoded image textures during viz compositing directly, that would benefit platforms beyond mac. Also avoid allocating textures for tiles.

- Decode directly into GMBs to avoid a texture upload. +Chris Blume had started work on this. Should work for Mac/CrOS and maybe also Android.

Indeed this would be ideal. Note that it wouldn't result in a net increase in memory usage.

This is the right way to hold it (from above).

[Vladimir Levin]

This also can apply to static images that are directly composited (similar idea in crbug.com/740743) 

[Dale Curtis]

Minor note: This runs the same risks it does for video for formats like AVIF (later frames need read access, so the GMBs can't be locked and slow read access may hurt performance), but at least GIF/WebP/APNG would be a benefit since every frame is a key frame. I don't recall if JPEG-XL has any intraframe compression.

[Khushal Sagar]

I'd say the lack of such an API is a limitation for any raster inducing animation. It's one thing where the property being animated can be composited (ideally by CoreAnimation), so layerizing an element allows us to skip raster. But if we have to re-raster for the animation anyway, then layerizing it is pure memory overhead. GIFs is the only special case where we need a staging buffer to decode the image so we can avoid allocating tiles. But for everything else, layerizing means using more tile textures and for no benefit other than to work around the CoreAnimation limitation.

[Jayson Adams]

Two questions: first, I want to trace the quads(?) from the renderer to the GPU process(?) and see how they result in these very wide update rects—is there a design doc/other form of documentation I can read to learn how all this works?

Second, there's also this case of a very wide update region for a simple flashing insertion point (visit this page and see attached screen capture). Is the page perhaps flashing an insertion point on top of an image? Or maybe this is the result of Mac CALayers and doesn't happen on other platforms? I'm trying to understand if there's perhaps a larger problem than how animated gifs work or if this login page example is a different problem entirely.

[Bo Liu]

During the exploration sprint earlier this year, some folks on the viz waterloo team looked into damage tracking. Internal presentation summarizing some findings: https://docs.google.com/presentation/d/1w9EHvz4UJ2KYG9DhBqZTEUvbPnZiKBUe2m6LcdueXLg/edit?usp=sharing&resourcekey=0-LsqNDn6_NrPtrs0GHzxjww

I explored the cc part:

* Each cc::CompositorFrame has a single rect of damage (on the root RenderPass). If a frame has smaller disjoint damages, they are combined in cc into the bounding rectangle.

* Ignoring the above, cc damage tracking is pretty good. Eg damage is not expanded to an entire tile. A blinking cursor damages very few pixels.

In viz, surface aggregation unions damage as well from the cc::CompositorFrame of tree of surfaces.

I forget how surface aggregation treats overlays, which in theory removes damage. IIR/UC Maggie worked on ensuring if an entire viz::Surface is lifted to an overlay (eg video), its damage can be removed and won't contribute to the root frame. However if a single quad of cc::CompositorFrame containing multiple quads is lifted to overlays, then the damage can't be removed, since it was already merged by cc.

Then damage from previous frames are merged in. Eg for triple buffering, the need to merge damage of the last frame as well.