Intent to ship: re-rastering composited layers on scale change

[Chris Harrelson]

Contact emails

chri...@chromium.org, dan...@chromium.org

Spec

The document describing this change is here (including a simple example). We believe that this change conforms to the intent and spirit of the will-change spec as a user-agent behavior, so no standards documentation is required.

Summary

All (composited) content will be re-rastered at the optimal (“ideal” in the nomenclature of the Chromium Compositor) scale when its transform scale changes, iff it does not have the will-change: transform CSS property.

Currently, composited layers raster their first frame at the scale you would expect (including transforms, etc). If that scale ever changes, it re-rasters at device scale, ignoring all transforms (*).

This change was discussed in some detail by a number of Blink rendering leads & engineers. A resulting detailed design document justifying this approach is here.

Link to “Intent to Implement” blink-dev discussion

N/A

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Demo link

Example.

Interoperability and Compatibility Risk

Other browsers do not implement this approach. Summary of what we know about their behavior:

• Gecko appears to re-raster composited layers when their scale changes.

• WebKit re-rasters with heuristics which are often similar to Blink’s current heuristic, but sometimes different. (We do not know the exact heuristics used. The example given in this intent renders blurry output for even the “crisp always” text, for example.)

• Edge re-rasters composited layers on scale change (I think; will double-check).

OWP launch tracking bug

crbug.com/619193

Entry on the feature dashboard

TODO (will add once this Intent is approved).

(*) The device scale behavior is chosen for historical reasons in order to match expectations of certain common websites far in the past. It really makes no sense in general.

[Chris Harrelson]

The primary risk of this Intent is worse performance on some sites. See crbug.com/600482 for some examples that happened when we tried to re-raster always without a developer escape-hatch. I have reached out to all developers who commented on that bug, as well as some Google-internal ones, with this new approach, which will require them to make small changes to their site to preserve performance.

We have also collected two UseCounters to verify assumptions that only a small percentage of pages are affected. The use counters are:

FixedRasterScaleBlurryContent: Triggers on a page that has content which is blurry today, but would be crisp after this change. Triggers on 0.5% of page loads on the beta channel.

FixedRasterScalePotentialPerformanceRegression: Triggers on a page that animates composited layer transform scales, and hence potentially has a performance regression after this change (and before adding will-change: transform to fix it). Triggers on 0.08% of page loads on the beta channel.

Note that a higher percentage for FixedRasterScaleBlurryContent is just fine, as those are simply pages we will be rendering with less-blurry content, which is pure win.

[PhistucK]

Hm, I feel like that this intent sort of builds on a bug. In Internet Explorer 11, I do not see any blurry text.

Why does Blink show blurry text in the first place?

And why is it acceptable for the first line to remain blurry?

If someone wants blurry text, they would use filter: blur(...);, or scale up a <canvas> (and other similar means).

Am I missing something?

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Chris Harrelson]

On Sat, Jun 11, 2016 at 2:40 AM PhistucK <phis...@gmail.com> wrote:

Hm, I feel like that this intent sort of builds on a bug. In Internet Explorer 11, I do not see any blurry text.

Why does Blink show blurry text in the first place?

And why is it acceptable for the first line to remain blurry?

This intent is about when we should decide to re-raster composited content that changes scale. Should we always re-raster? Never unless asked? After a timeout? When possible? The fancier the solution, the more code complexity, and potentially magic/unintended behavior for the developer who wants to implement a curated animation. For example, if we re-raster at high quality after a timeout, that will cause pop in the visible UI that the developer may not want. It will also cause scheduling challenges for chrome, since we would need to avoid doing useless work more often, as well as lots of code complexity.

Note that non-composited content always re-rasters on every animation frame if something changed. Composited layers are special, in that they were designed for performance and taking advantage of GPUs. Unfortunately the current focus on performance yields a lot of ugly content, and content that in some cases content that remains blurry without any reasonable recourse. Hence this intent to address that.

This intent proposes removing all heuristics and either rastering everything on all animation frames (by default), or not at all (opt-in via will-change: transform). We think this strikes a good balance of a non-blurry and high-quality output by default, but control of when raster happens available to developers who want fast animation or to avoid undesireable certain side-effects of rastering on every frame (e.g. animations looking strange due to aliasing). And minimizes the code complexity of Chromium.

[Rick Byers]

Thank you for making this more predictable!  In general this seems more in-line with what developers would naively expect, and what Edge and Firefox are doing.  I agree the UMA suggests this is worth trying.

If I understand correctly, will-change: transform today and after this change forces content to be rastered at scale 1, right?  I.e. there's no risk that this change will cause content that was previously crisp to become blurry.

LGTM1

[Chris Harrelson]

On Mon, Jun 13, 2016 at 6:32 AM Rick Byers <rby...@chromium.org> wrote:

Thank you for making this more predictable!  In general this seems more in-line with what developers would naively expect, and what Edge and Firefox are doing.  I agree the UMA suggests this is worth trying.

If I understand correctly, will-change: transform today and after this change forces content to be rastered at scale 1, right?  I.e. there's no risk that this change will cause content that was previously crisp to become blurry.

No, it rasters at the scale the composited layer is at when it is created (including transforms, etc). This is a little bit different than the current behavior which is:

a. First raster at scale when created

b. If the scale ever changes, raster at device scale ("scale 1")

(b) will be going away, since it's an arbitrary heuristic. Therefore there will be a small number of sites that happened to benefit from (b) which will look a little bit worse. 

Chris

[Rick Byers]

On Mon, Jun 13, 2016 at 12:11 PM, Chris Harrelson <chri...@chromium.org> wrote:

On Mon, Jun 13, 2016 at 6:32 AM Rick Byers <rby...@chromium.org> wrote:

Thank you for making this more predictable!  In general this seems more in-line with what developers would naively expect, and what Edge and Firefox are doing.  I agree the UMA suggests this is worth trying.

If I understand correctly, will-change: transform today and after this change forces content to be rastered at scale 1, right?  I.e. there's no risk that this change will cause content that was previously crisp to become blurry.

No, it rasters at the scale the composited layer is at when it is created (including transforms, etc). This is a little bit different than the current behavior which is:

a. First raster at scale when created

b. If the scale ever changes, raster at device scale ("scale 1")

(b) will be going away, since it's an arbitrary heuristic. Therefore there will be a small number of sites that happened to benefit from (b) which will look a little bit worse. 

Ah, I see.  This seems like an important point to cover in the doc (we may need to write some guidance for developers, etc.).  What makes you say this is "small"?  Eg. today if I want to animate something in that starts small and smoothly grows to it's desired side, it's reasonable that I will have used will-change:transform, right?

What's the recommended replacement for that - where I want to raster only once at the target size (to avoid really blurry intermediate frames)?  Do I start at "transform: scale(1); opacity:0; will-change: transform", then scale down and remove the transparency, then animate the scale back up to 1?

[Chris Harrelson]

On Mon, Jun 13, 2016 at 9:40 AM Rick Byers <rby...@chromium.org> wrote:

On Mon, Jun 13, 2016 at 12:11 PM, Chris Harrelson <chri...@chromium.org> wrote:

On Mon, Jun 13, 2016 at 6:32 AM Rick Byers <rby...@chromium.org> wrote:

Thank you for making this more predictable!  In general this seems more in-line with what developers would naively expect, and what Edge and Firefox are doing.  I agree the UMA suggests this is worth trying.

If I understand correctly, will-change: transform today and after this change forces content to be rastered at scale 1, right?  I.e. there's no risk that this change will cause content that was previously crisp to become blurry.

No, it rasters at the scale the composited layer is at when it is created (including transforms, etc). This is a little bit different than the current behavior which is:

a. First raster at scale when created

b. If the scale ever changes, raster at device scale ("scale 1")

(b) will be going away, since it's an arbitrary heuristic. Therefore there will be a small number of sites that happened to benefit from (b) which will look a little bit worse. 

Ah, I see.  This seems like an important point to cover in the doc (we may need to write some guidance for developers, etc.).  What makes you say this is "small"?

We don't have this data unfortunately, so it's an educated guess about the percentage of time relative to the FixedRasterScaleBlurryContent use counter. The current heuristic will be superior generally only if the content animates towards its device-scale size, and even then, will not necessarily be that much better unless it ends up with exactly device-scale 1.

The guidance already in the one-pager I wrote and linked to in this thread addresses this use case, as well as the more general situation. A developer tutorial should go into more detail though, I agree.

 

  Eg. today if I want to animate something in that starts small and smoothly grows to it's desired side, it's reasonable that I will have used will-change:transform, right? 

Developers should only do that if they are sure that it really is difficult to raster on every frame. If you're animating icons or anything reasonably small, it's often better to raster on every frame, and in fact not to have a composited layer at all if possible.  

What's the recommended replacement for that - where I want to raster only once at the target size (to avoid really blurry intermediate frames)?  Do I start at "transform: scale(1); opacity:0; will-change: transform", then scale down and remove the transparency, then animate the scale back up to 1?

You'll get blurry intermediate frames even today, it'll just look crisp at the end if the animation ends at device scale. One possible approach is what you outlined (teach the composited layer about the desired bitmap scale at start of animation), and that is in fact what Chromium does for CSS animations on behalf of developers, since we know the animation curve in advance. (Aside: this intent only applies to javascript-driven animations, not CSS animations. I'll add a note about that to the doc.

The example you gave is indeed difficult to express, and is a use case where the current heuristic does better. Room for potential improvement in the future.

Specifying opacity: 0 won't work because that affects the rastered output.

[TAMURA, Kent]

LGTM2.

I think this improves interoperability and predictability.

--

TAMURA Kent
Software Engineer, Google

[Philip Jägenstedt]

LGTM3, but the lack of control over the initial scale when using will-change: transform is unfortunate. Do you think that always using the size ignoring transforms would be terrible, and if so how about using the largest of the size with transforms and the size ignoring transforms? Of course, however it's done one can end up with blurry content, but without new CSS syntax it seems like the size ignoring transforms is the only other signal there is.

[Elliott Sprehn]

On Mon, Jun 20, 2016 at 3:53 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

LGTM3, but the lack of control over the initial scale when using will-change: transform is unfortunate. Do you think that always using the size ignoring transforms would be terrible, and if so how about using the largest of the size with transforms and the size ignoring transforms? Of course, however it's done one can end up with blurry content, but without new CSS syntax it seems like the size ignoring transforms is the only other signal there is.

In coordination with shipping this we should really file a bug with the CSSWG that an initial-scale CSS property is needed to control it. 

- E 

[Chris Harrelson]

Thanks all. CL in progress.

On Mon, Jun 20, 2016 at 8:30 AM, Elliott Sprehn <esp...@chromium.org> wrote:

On Mon, Jun 20, 2016 at 3:53 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

LGTM3, but the lack of control over the initial scale when using will-change: transform is unfortunate. Do you think that always using the size ignoring transforms would be terrible, and if so how about using the largest of the size with transforms and the size ignoring transforms? Of course, however it's done one can end up with blurry content, but without new CSS syntax it seems like the size ignoring transforms is the only other signal there is.

Yes this would help with the particular use case of needing to raster at device scale. But (a) there are various cases where that is not a good thing to do (e.g. animating to something other than device scale) (in which case the output would have worse pixel quality due to downsampling), and (b) this uses significantly more memory if the current scale vs device scale is very different.

I'll also repeat for the record that the developer can in fact get the good rastering at device scale after animating up, at the cost of a second raster (Rick mentioned this, but to be extra clear...).

 

In coordination with shipping this we should really file a bug with the CSSWG that an initial-scale CSS property is needed to control it. 

Ok I'll do that.