Intent to Implement: javascript image decode() api.

[Vladimir Levin]

Contact emails

vmp...@chromium.org, dom...@chromium.org

Spec

Spec proposal is here:

https://github.com/whatwg/html/pull/2332

The spec has been reviewed and is considered ready to merge by the editors but we'd like sign off from WebKit before performing the merge.

Summary

This change allows web developers to request to decode an img element. The call to a new HTML <img> element’s decode() function returns a promise, which, when fulfilled, ensures that the image can be appended to the DOM without causing a decoding delay on the next frame.

Sample code:

var img = new Image();

img.src = "nebula.jpg";

img.decode().then(function() {

 document.getElementById("container").appendChild(img);

});

Motivation

Currently there is no simple way for the web developer to insert an image into the DOM without causing a decoding delay.

You can see a demo of a prototype here:

https://drive.google.com/file/d/0B5cKk7MxQ2LsUUotR3RuWUlpSXc/view?usp=sharing

The first page, sync.html, uses an onload handler, which when fired will append the image to the DOM. This causes a noticeable delay in the requestAnimationFrame-driven animation of the spinning arrow.

The second page, async.html, uses decode() functionality (similar to the sample code above), which when resolved will append the image to the DOM. Note a significant reduction in the delay during the raf animation of the spinning arrow.

The video was made using a prototype patch (still work in progress):

https://codereview.chromium.org/2769823002/

Interoperability and Compatibility Risk

This feature has interest and support from WebKit, who initiated the discussion. There are no signals from Edge or Gecko at this time.

As for compatibility: as long as web developers feature-detect for this API, it can be removed without risk. If they do not, then removing it could break the image display on pages using code similar to the sample above, or cause other program flow errors. This could be mitigated by replacing it with a no-op method that always returns a fulfilled promise.

Discussion links:

Initial proposal: https://github.com/whatwg/html/issues/1920

More discussion: https://github.com/whatwg/html/issues/2037

Ongoing technical constraints

None. As mentioned, this is a promise based approach which means that the work to decode the image could happen asynchronously on any thread. As long as the promise is satisfied with correct behavior results, the way the work gets done is up to the UA.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Requesting approval to ship?

No

[ben.m...@gmail.com]

It sounds like there was a lot of discussion on this thread about an async attribute for images which would have ensured that the decoding never blocked the main thread. It sounds like UAs have some flexibility to implement that behavior without a spec change.

Any thoughts about when the browser should be proactively doing this vs a developer using this kind of API.

[Vladimir Levin]

> Can you create a chromestatus.com entry for this? Would love to share with developers!

I just filed https://www.chromestatus.com/feature/5637156160667648

> It sounds like there was a lot of discussion on this thread about an async attribute for images which

> would have ensured that the decoding never blocked the main thread. It sounds like UAs have some

> flexibility to implement that behavior without a spec change.

>

> Any thoughts about when the browser should be proactively doing this vs a developer using this kind of API.

That's a good question. You're right that UAs could prevent the decoding delay by itself, but it's unclear in what situations it would appropriate for it to do that. For example, it may be important that the image appears together with non image content, which makes it not suitable for the UA to delay displaying the image. I think there may be a heuristic that works there and we're looking into this approach as well. The async attribute would help solve this heuristic by becoming a hint to the UA that it's OK/not-OK to display the image whenever it's ready (asynchronously).

This decode API, however, is meant to provide that control to the developer. That is, it is up to the developer to request this decode and inject it wherever needed when it's done; the UA, on the other hand, promises to prepare as much of the image as possible to ensure that displaying the image is as quick as possible. This makes it possible to synchronize displaying image and non image content without decoding delay.

I hope that answers your question.

[Ben Maurer]

I see. It seems like there's two scenarios here:

(1) I listen to the load event and assert expect my content to show up in exactly that frame.

(2) I just put the image into the dom knowing that there will be network / decoding delay

I wonder if there's any way to make the experience in (2) easier. As an example, if you have an <img> tag that is in the DOM and that image is visible, don't fire the load event until the image has been decoded. You know that the image will be displayed based on the position of the tag.

From a developer perspective it seems like getting some good heuristics in here would avoid the need for a lot of annotation.

[Vladimir Levin]

For (1), If you have an image that is out of the DOM, listening to the load event is typically not a strong enough signal to ensure that that the image will not cause delay when inserted into the DOM (the demo video in the proposal is exactly this case, comparing load vs decode in Chrome). That's the case targeted by the proposed API.

For (2), I think the discussion about the async attribute in one of the discussion links talks about this. Specifically, it talks about annotating images as async, which means you (the UA) don't have to have this displayed when I put it in the DOM, rather make sure the decode doesn't block other things going on.

As an aside, a similar test to the one in the videos has different results on different UAs. (test is: we have a raf animation going, we create an image and onload we attach the image and some text):

1. Chrome and Safari -- the image and text appear at the same time with a noticeable delay in the raf animation

2. Edge and Firefox -- the text appears before the image, the image only appears roughly after the same time as as the delay in 1., but there is no actual delay in the animation. 

So it seems like UAs in 2. already have a heuristic in place that delays displaying the images until they are decoded, but the decode doesn't correspond to the onload event (hence the images and text appear at different times)... I do think that this is worth investigating more, but at the same time I feel like the decode api proposed here addresses a slightly different use case. What do you think?

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Ben Maurer]

If there are differences between UAs here I think it might make sense to investigate making the default the edge/ff behavior where image decoding is async by default. In most cases developers aren't synchronizing the appearance of text and images. It's not even clear to me that doing so is a best practice in most cases since images might be slow due to network. If this were the default, the decode() api would be the escape hatch for developers to get the behavior that exists today. makes sense how these fit together.

[Domenic Denicola]

From: blin...@chromium.org [mailto:blin...@chromium.org] On Behalf Of ben.m...@gmail.com

> It sounds like there was a lot of discussion on this thread about an async attribute for images which would have ensured that the decoding never blocked the main thread. It sounds like UAs have some flexibility to implement that behavior without a spec change.
>
> Any thoughts about when the browser should be proactively doing this vs a developer using this kind of API.

There was indeed a lot of good discussion. We ended up splitting the discussion into focusing on two separate use cases, deferring and predecoding. See https://github.com/whatwg/html/issues/1920#issuecomment-256114499 for more detail.

This API, image.decode(), is focused on the predecoding use case. There is still value in addressing the deferring use case for when developers do not want explicit control, and we'll probably be working on that in the future as well.

Where we left that discussion is in https://github.com/whatwg/html/issues/1920#issuecomment-259763168:

> It seems like the biggest outstanding point for the attribute to decide on is whether this is a two- or three-state attribute. Does anyone want to help further drive the discussion on that?

[Joe Medley]

Vladimir,

Do you have any kind of tracking bug, OWP or otherwise, I can use to follow the progress of this?

On Monday, March 27, 2017 at 11:58:21 AM UTC-7, Vladimir Levin wrote:

[Vladimir Levin]

Hey, there is a crbug.com/705669 that is tracking this. I will update it shortly with links to the spec draft as well.