Intent to Ship: image decoding attribute

1143 views
Skip to first unread message

Vladimir Levin

unread,
Dec 4, 2017, 5:40:44 PM12/4/17
to blink-dev, Chris Harrelson, Domenic Denicola

Contact emails

vmp...@chromium.org, chri...@chromium.org


Explainer

https://docs.google.com/document/d/1VvrEGAjcnto5egR_rnX7PR5ymqWayMNRPYSbCf1bC4I/edit#


Spec

Spec PR: https://github.com/whatwg/html/pull/3221

Tag review: https://github.com/w3ctag/design-reviews/issues/220


Summary

We propose to introduce a decoding attribute on HTMLImageElement and SVGImageElement which would act as a hint to Chrome. This attribute would have three states:

  • “async”: This indicates that the developer prefers responsiveness and performance over atomic presentation of content.

  • “sync”: This indicates that the developer prefers atomic presentation of content over responsiveness.

  • "auto": This indicates no preference from the developer, and acts as the default value.


Link to “Intent to Implement” blink-dev discussion

https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/9i6wgXv7c7c


Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.


Risks: Interoperability and Compatibility

Since this attribute acts as a hint, there is little risk in interoperability. Specifically, if other browsers do not implement it, they would not be necessarily violating the spec. Additionally, since this is an attribute browsers may decide not to parse this value at all with no detriment to either the developer or the user.


If we need to remove this feature, it would also be straight forward in that content may still use the attribute but Chrome can silently ignore it.


Edge: Support.

Firefox: Support.

Safari: In development

Web developers: Positive (AMP team indicated desire to start using the attribute)


See discussion on https://github.com/whatwg/html/issues/1920

Risks: Ergonomics
Since the attribute is a hint, there is little risk of ergonomics interop problems.

Risks: Activation
It should be straightforward for developers to use this feature.

Alex Russell

unread,
Dec 7, 2017, 1:42:39 PM12/7/17
to blink-dev, chri...@chromium.org, dom...@chromium.org
Hey all,

Thanks for filing a TAG review issue. As we haven't had a chance to discuss it there yet, wanted to understand the feature more deeply:
  • How does this interact with other raster-time behaviors? I.e., will a "sync" value delay the start of an animation which happens to contain the image?
  • Should this be specifiable on a parent element instead of just leaf-images?
  • Should this be something you can query from CSS? How is it represented in computedStyle (if at all)?
  • How does a developer coordinate multiple images presenting together? Is there any sort of "grouping" concept planned?
  • For developers that want to introduce a fade-in style animation for image content once it arrives, does this make it easier? harder? neutral?
  • How does this interact with scrolling and lazy-loading? If I specify this attribute on an image that's off-screen, will (or should) it delay painting tiles for that stuff? Should it cause the engine to fetch that content more eagerly?
  • Is there any memory impact on a site that specifies this for many images?
Regards

Vladimir Levin

unread,
Dec 7, 2017, 2:19:11 PM12/7/17
to Alex Russell, blink-dev, Chris Harrelson, Domenic Denicola
On Thu, Dec 7, 2017 at 10:42 AM, 'Alex Russell' via blink-dev <blin...@chromium.org> wrote:
Hey all,

Thanks for filing a TAG review issue. As we haven't had a chance to discuss it there yet, wanted to understand the feature more deeply:
  • How does this interact with other raster-time behaviors? I.e., will a "sync" value delay the start of an animation which happens to contain the image?
"sync" will ensure that the image is rasterized along with the other content that appears in that update. So in terms of animation, yes it will ensure that the image is rasterized before the animation begins. Note that I'm a little weary about using the term delay here since the current Chrome implementation would have the same delay. If we move more towards having async as default, the start of the animation would theoretically happen faster and when that happens, then the "sync" value would be the opt-out of this behavior for the sake of having the image there when the animation starts. In other words the delay would bring us back to where we are now.
 
  • Should this be specifiable on a parent element instead of just leaf-images?
This should only be specifiable directly on leaf images. It seems a bit awkward to specify "decoding" on an element that isn't an image, but may contain images.
 
  • Should this be something you can query from CSS? How is it represented in computedStyle (if at all)?
I believe there's a separate proposal (in the early stages) to add a CSS property to deal with things like background images and allow them to be asynchronous. The attribute proposed here would not be in CSS or computedStyle.
 
  • How does a developer coordinate multiple images presenting together? Is there any sort of "grouping" concept planned?
In order to present multiple images together, first the developer would have to have onload handlers on the images to ensure all of the images are available. Then to avoid images popping in at various times due do different decoding times, the developer would specify decoding=sync. We don't have a grouping concept planned.
 
  • For developers that want to introduce a fade-in style animation for image content once it arrives, does this make it easier? harder? neutral?
I would say it's more or less neutral. Some user-agents may already have "async" default behavior, and the fade in at low alpha is typically long enough for the image to appear without the user noticing. So, not specifying the attribute would continue to have this behavior. With this attribute, however, we could still specify decoding=sync to ensure that the image is rasterized right at the start of the fade-in.
 
  • How does this interact with scrolling and lazy-loading? If I specify this attribute on an image that's off-screen, will (or should) it delay painting tiles for that stuff? Should it cause the engine to fetch that content more eagerly?
If async is specified for an off-screen image, then it is plausible that the browser will not rasterize that image when it first enters the viewport and it will appear shortly after. This greatly improves performance for a fast fling on image heavy sites. That is, if the image is in the viewport one frame, we can defer it. Then if the next frame the image exits the viewport, in some cases we can avoid decoding it altogether and start working on the new content that is now in the viewport. 

I'm not sure what you mean by the fetch part of the question. I think as far as downloading the image content, this attribute doesn't affect it. It only affects what happens after we have the downloaded image and we need to rasterize it. "sync" would mean we rasterize it right there and then with other content. "async" would mean that we can defer the image rasterization and instead present a frame with the rest of the non-image content while decoding the image off the critical path. 
 
  • Is there any memory impact on a site that specifies this for many images?
There shouldn't be any memory impact with this feature. All of the image decoding already happens in current browsers. Because of the large size of decoded images, browsers already implement smart caching mechanisms that manage and evict memory as needed when image decodes aren't needed any more. The use of this attribute provides a hint as to when to decode the image (on the critical path vs not), but it doesn't mandate any memory management, so I would expect that browsers would continue to manage the decodes the way they do now. 
 
Thanks for the detailed questions! I hope my answers clarify this a bit, but let me know if you have any follow-up questions
Vlad

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7b95566f-f8a8-4090-bc82-c480978dead7%40chromium.org.

Chris Harrelson

unread,
Dec 7, 2017, 2:33:43 PM12/7/17
to Vladimir Levin, Alex Russell, blink-dev, Domenic Denicola
On Thu, Dec 7, 2017 at 11:18 AM, Vladimir Levin <vmp...@chromium.org> wrote:
  • How does this interact with scrolling and lazy-loading? If I specify this attribute on an image that's off-screen, will (or should) it delay painting tiles for that stuff? Should it cause the engine to fetch that content more eagerly?

I'm not sure what you mean by the fetch part of the question. I think as far as downloading the image content, this attribute doesn't affect it. It only affects what happens after we have the downloaded image and we need to rasterize it. "sync" would mean we rasterize it right there and then with other content. "async" would mean that we can defer the image rasterization and instead present a frame with the rest of the non-image content while decoding the image off the critical path. 

One more point that I think is relevant: this attribute does not impact the ability of the browser to avoid rastering or decoding offscreen content. Chrome (and almost certainly all other browsers) already has lots of code to predict when offscreen content might come onto the screen, and pre-decodes and pre-rasterizes content. Regardless of the decoding attribute, we will decode images that are predicted to come on screen, though the value of the attribute will inform task priority, and of course whether the image must present to the screen at the same time as surrounding content.

A second point to make is that Chrome (and likely other browsers) has a tiling architecture for representing content to display on the screen, and rasters those tiles independently. In the case of composited scrolling, Chrome already prioritizes scroll performance over atomically presented content, which can result in some tiles not being visible at the same time as others; this feature is called checkerboarding. The decoding attribute and the decoding=async attribute specifically will allow us to reduce the cost of tiles and therefore reduce checkerboarding, by displaying content without images. Other than that, tile checkerboarding and scrolling is not affected by the decoding attribute.

Chris
 

Alex Russell

unread,
Dec 13, 2017, 2:49:04 PM12/13/17
to Vladimir Levin, blink-dev, Chris Harrelson, Domenic Denicola
Sorry for the slow reply (and for the subsequent response to Chris's message because threading and truncation).

On Thu, Dec 7, 2017 at 11:18 AM, Vladimir Levin <vmp...@chromium.org> wrote:


On Thu, Dec 7, 2017 at 10:42 AM, 'Alex Russell' via blink-dev <blin...@chromium.org> wrote:
Hey all,

Thanks for filing a TAG review issue. As we haven't had a chance to discuss it there yet, wanted to understand the feature more deeply:
  • How does this interact with other raster-time behaviors? I.e., will a "sync" value delay the start of an animation which happens to contain the image?
"sync" will ensure that the image is rasterized along with the other content that appears in that update. So in terms of animation, yes it will ensure that the image is rasterized before the animation begins. Note that I'm a little weary about using the term delay here since the current Chrome implementation would have the same delay. If we move more towards having async as default, the start of the animation would theoretically happen faster and when that happens, then the "sync" value would be the opt-out of this behavior for the sake of having the image there when the animation starts. In other words the delay would bring us back to where we are now.

I see. Thanks! 
  • Should this be specifiable on a parent element instead of just leaf-images?
This should only be specifiable directly on leaf images. It seems a bit awkward to specify "decoding" on an element that isn't an image, but may contain images.

It also seems awkward to need to set attributes on many images when you want a more global behavior. Have we considered making this applicable via CSS instead? 
  • Should this be something you can query from CSS? How is it represented in computedStyle (if at all)?
I believe there's a separate proposal (in the early stages) to add a CSS property to deal with things like background images and allow them to be asynchronous. The attribute proposed here would not be in CSS or computedStyle.

That's...odd. Having some images have this behaviour set via attributes and others via CSS (with no unified query semantic) seems like are recipe for confusion.
 
  • How does a developer coordinate multiple images presenting together? Is there any sort of "grouping" concept planned?
In order to present multiple images together, first the developer would have to have onload handlers on the images to ensure all of the images are available. Then to avoid images popping in at various times due do different decoding times, the developer would specify decoding=sync. We don't have a grouping concept planned.

I'd like to see sample code for presenting multiple images together. Does that exist?
 
  • For developers that want to introduce a fade-in style animation for image content once it arrives, does this make it easier? harder? neutral?
I would say it's more or less neutral. Some user-agents may already have "async" default behavior, and the fade in at low alpha is typically long enough for the image to appear without the user noticing. So, not specifying the attribute would continue to have this behavior. With this attribute, however, we could still specify decoding=sync to ensure that the image is rasterized right at the start of the fade-in.

Developers are frequently implementing low-res/high-res transitions and cross-faded with expensive methods (see, e.g., Polymer Shop or Medium). It seems like this behavior has an overlap with that use-case and I'd like to understand how developers can get better control.
 
  • How does this interact with scrolling and lazy-loading? If I specify this attribute on an image that's off-screen, will (or should) it delay painting tiles for that stuff? Should it cause the engine to fetch that content more eagerly?
If async is specified for an off-screen image, then it is plausible that the browser will not rasterize that image when it first enters the viewport and it will appear shortly after.

Will this flicker/snap?
 
This greatly improves performance for a fast fling on image heavy sites. That is, if the image is in the viewport one frame, we can defer it. Then if the next frame the image exits the viewport, in some cases we can avoid decoding it altogether and start working on the new content that is now in the viewport. 

I'm not sure what you mean by the fetch part of the question. I think as far as downloading the image content, this attribute doesn't affect it. It only affects what happens after we have the downloaded image and we need to rasterize it. "sync" would mean we rasterize it right there and then with other content. "async" would mean that we can defer the image rasterization and instead present a frame with the rest of the non-image content while decoding the image off the critical path. 

Thanks for the explanation! 
  • Is there any memory impact on a site that specifies this for many images?
There shouldn't be any memory impact with this feature. All of the image decoding already happens in current browsers. Because of the large size of decoded images, browsers already implement smart caching mechanisms that manage and evict memory as needed when image decodes aren't needed any more. The use of this attribute provides a hint as to when to decode the image (on the critical path vs not), but it doesn't mandate any memory management, so I would expect that browsers would continue to manage the decodes the way they do now. 
 
Thanks for the detailed questions!

Appreciate the thorough reply!

Regards

Alex Russell

unread,
Dec 13, 2017, 2:51:17 PM12/13/17
to Chris Harrelson, Vladimir Levin, blink-dev, Domenic Denicola
Hey Chris,

Inline:

On Thu, Dec 7, 2017 at 11:33 AM, Chris Harrelson <chri...@chromium.org> wrote:


On Thu, Dec 7, 2017 at 11:18 AM, Vladimir Levin <vmp...@chromium.org> wrote:
  • How does this interact with scrolling and lazy-loading? If I specify this attribute on an image that's off-screen, will (or should) it delay painting tiles for that stuff? Should it cause the engine to fetch that content more eagerly?

I'm not sure what you mean by the fetch part of the question. I think as far as downloading the image content, this attribute doesn't affect it. It only affects what happens after we have the downloaded image and we need to rasterize it. "sync" would mean we rasterize it right there and then with other content. "async" would mean that we can defer the image rasterization and instead present a frame with the rest of the non-image content while decoding the image off the critical path. 

One more point that I think is relevant: this attribute does not impact the ability of the browser to avoid rastering or decoding offscreen content. Chrome (and almost certainly all other browsers) already has lots of code to predict when offscreen content might come onto the screen, and pre-decodes and pre-rasterizes content. Regardless of the decoding attribute, we will decode images that are predicted to come on screen, though the value of the attribute will inform task priority, and of course whether the image must present to the screen at the same time as surrounding content.

A second point to make is that Chrome (and likely other browsers) has a tiling architecture for representing content to display on the screen, and rasters those tiles independently. In the case of composited scrolling, Chrome already prioritizes scroll performance over atomically presented content, which can result in some tiles not being visible at the same time as others; this feature is called checkerboarding. The decoding attribute and the decoding=async attribute specifically will allow us to reduce the cost of tiles and therefore reduce checkerboarding, by displaying content without images. Other than that, tile checkerboarding and scrolling is not affected by the decoding attribute.

Vlad mentioned a potential future where we switch the default policy. Is the hope that this attribute will unblock such a change, or is that orthogonal?

Regards

Chris Harrelson

unread,
Dec 13, 2017, 2:54:24 PM12/13/17
to Alex Russell, Vladimir Levin, blink-dev, Domenic Denicola
On Wed, Dec 13, 2017 at 11:50 AM, 'Alex Russell' via blink-dev <blin...@chromium.org> wrote:
Hey Chris,

Inline:

On Thu, Dec 7, 2017 at 11:33 AM, Chris Harrelson <chri...@chromium.org> wrote:


On Thu, Dec 7, 2017 at 11:18 AM, Vladimir Levin <vmp...@chromium.org> wrote:
  • How does this interact with scrolling and lazy-loading? If I specify this attribute on an image that's off-screen, will (or should) it delay painting tiles for that stuff? Should it cause the engine to fetch that content more eagerly?

I'm not sure what you mean by the fetch part of the question. I think as far as downloading the image content, this attribute doesn't affect it. It only affects what happens after we have the downloaded image and we need to rasterize it. "sync" would mean we rasterize it right there and then with other content. "async" would mean that we can defer the image rasterization and instead present a frame with the rest of the non-image content while decoding the image off the critical path. 

One more point that I think is relevant: this attribute does not impact the ability of the browser to avoid rastering or decoding offscreen content. Chrome (and almost certainly all other browsers) already has lots of code to predict when offscreen content might come onto the screen, and pre-decodes and pre-rasterizes content. Regardless of the decoding attribute, we will decode images that are predicted to come on screen, though the value of the attribute will inform task priority, and of course whether the image must present to the screen at the same time as surrounding content.

A second point to make is that Chrome (and likely other browsers) has a tiling architecture for representing content to display on the screen, and rasters those tiles independently. In the case of composited scrolling, Chrome already prioritizes scroll performance over atomically presented content, which can result in some tiles not being visible at the same time as others; this feature is called checkerboarding. The decoding attribute and the decoding=async attribute specifically will allow us to reduce the cost of tiles and therefore reduce checkerboarding, by displaying content without images. Other than that, tile checkerboarding and scrolling is not affected by the decoding attribute.

Vlad mentioned a potential future where we switch the default policy. Is the hope that this attribute will unblock such a change, or is that orthogonal?

This proposal unblocks that change, because it allows us to change the default behavior while having an opt-out for developers. So in fact I think this change will be performance-positive, because we intend to pair it with more async-by-default. It also increases interop because changing the default to be more async will move towards the current behavior in Gecko and Edge.

Chirs


Rick Byers

unread,
Dec 13, 2017, 3:26:54 PM12/13/17
to Chris Harrelson, Alex Russell, Vladimir Levin, blink-dev, Domenic Denicola
Personally I think there's sufficient consensus and low enough risk that we should ship this - so LGTM3

We should, of course, continue to discuss the details and refine as necessary.

Chris Harrelson

unread,
Dec 13, 2017, 3:38:01 PM12/13/17
to Alex Russell, Vladimir Levin, blink-dev, Domenic Denicola
On Wed, Dec 13, 2017 at 11:48 AM, Alex Russell <sligh...@chromium.org> wrote:
Sorry for the slow reply (and for the subsequent response to Chris's message because threading and truncation).

On Thu, Dec 7, 2017 at 11:18 AM, Vladimir Levin <vmp...@chromium.org> wrote:
  • Should this be specifiable on a parent element instead of just leaf-images?
This should only be specifiable directly on leaf images. It seems a bit awkward to specify "decoding" on an element that isn't an image, but may contain images.

It also seems awkward to need to set attributes on many images when you want a more global behavior. Have we considered making this applicable via CSS instead?

The decoding attributes is developer hints about rendering dependencies related to the element, which is why it's on the element and not CSS. Similar to the async keywords on scripts etc. The src attribute is similar also.
 
 
  • Should this be something you can query from CSS? How is it represented in computedStyle (if at all)?
I believe there's a separate proposal (in the early stages) to add a CSS property to deal with things like background images and allow them to be asynchronous. The attribute proposed here would not be in CSS or computedStyle.

That's...odd. Having some images have this behaviour set via attributes and others via CSS (with no unified query semantic) seems like are recipe for confusion.

The proposal for background images involves the proposed image() function. You could think of that as an image within the CSS. For background images, src etc are in CSS also.
 
 
  • How does a developer coordinate multiple images presenting together? Is there any sort of "grouping" concept planned?
In order to present multiple images together, first the developer would have to have onload handlers on the images to ensure all of the images are available. Then to avoid images popping in at various times due do different decoding times, the developer would specify decoding=sync. We don't have a grouping concept planned.

I'd like to see sample code for presenting multiple images together. Does that exist?
 
  • For developers that want to introduce a fade-in style animation for image content once it arrives, does this make it easier? harder? neutral?
I would say it's more or less neutral. Some user-agents may already have "async" default behavior, and the fade in at low alpha is typically long enough for the image to appear without the user noticing. So, not specifying the attribute would continue to have this behavior. With this attribute, however, we could still specify decoding=sync to ensure that the image is rasterized right at the start of the fade-in.

Developers are frequently implementing low-res/high-res transitions and cross-faded with expensive methods (see, e.g., Polymer Shop or Medium). It seems like this behavior has an overlap with that use-case and I'd like to understand how developers can get better control.

For the Medium example, they would use decode=sync to avoid flicker beyond the sequence they describe of low-res -> canvas + blur -> image. This preserves the behavior they have today. They could also add in the image decode API if they were worried about decode times for the particular image blocking display.

The shop app uses the same technique, but a CSS animation on opacity rather than canvas with blur, and never remove the low-res image. The same advice applies. In the Shop case, they could even put decode=async
right away, and the high-res image might display sometime during the opacity animation curve (currently 0.5s in Shop), which is a reasonable UI.

 
  • How does this interact with scrolling and lazy-loading? If I specify this attribute on an image that's off-screen, will (or should) it delay painting tiles for that stuff? Should it cause the engine to fetch that content more eagerly?
If async is specified for an off-screen image, then it is plausible that the browser will not rasterize that image when it first enters the viewport and it will appear shortly after.

Will this flicker/snap?

The image will come in later. In the first frame user sees, the image will not be present and instead the content behind it will be shown. Then the image will be shown. This is "flickering/snap" in that sense. In fact that's kind of the point of the feature - to prefer fast display over no-flickering.
 

Rick Byers

unread,
Dec 13, 2017, 3:39:45 PM12/13/17
to Chris Harrelson, Alex Russell, Vladimir Levin, blink-dev, Domenic Denicola
On Wed, Dec 13, 2017 at 3:26 PM, Rick Byers <rby...@chromium.org> wrote:
Personally I think there's sufficient consensus and low enough risk that we should ship this - so LGTM3

Correction - sorry, LGTM1. 

Was thinking of another intend thread where I was the hold out ;-).  The comments still refer to this intent though - just the count was wrong.

Philip Jägenstedt

unread,
Dec 14, 2017, 8:55:22 AM12/14/17
to Rick Byers, Chris Harrelson, Alex Russell, Vladimir Levin, blink-dev, Domenic Denicola
I tried for a while, but couldn't find anything to worry about. LGTM2 :)

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_p61oUzht5dQArouHJZ%3DxwFjF71VnyfN07Ncd%2B_CdxwA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

Ojan Vafai

unread,
Dec 14, 2017, 12:33:44 PM12/14/17
to Philip Jägenstedt, Rick Byers, Chris Harrelson, Alex Russell, Vladimir Levin, blink-dev, Domenic Denicola
Reply all
Reply to author
Forward
0 new messages