Intent to Implement: Support for Image Replacement in Blink

374 views
Skip to first unread message

Scott Little

unread,
Aug 17, 2016, 5:02:44 PM8/17/16
to blink-dev, Ben Greenstein, Tal Oppenheimer, Ojan Vafai, Kenji Baheux

Contact emails

scli...@chromium.org, be...@chromium.org


Spec

Design doc: https://docs.google.com/document/d/1691W7yFDI1FJv69N2MEtaSzpnqO2EqkgGD3T0O-pQ08/edit#


Summary

Add support to Blink for removing or postponing image loads while preserving page layout. To achieve this, support issuing range requests for just the first few bytes of images, and then attempt to extract the image dimensions from these chunks and display placeholders with the same dimensions in their place.


This support will not be activated by default, and there will be no impact to the regular path of loading images.


Motivation

Allow other features to activate this support in order to save data for users.


Potential use cases of this functionality in Chromium include:

  • Replace images with placeholders on heavy pages or extremely slow networks for Data Saver users, similar to Lo-Fi
  • Defer loading of below-the-fold images until the user scrolls down to them

Interoperability and Compatibility Risk

No risk.


Ongoing technical constraints

None.


Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.


OWP launch tracking bug

No OWP launch tracking bug, since this is just adding low-level support that will be used by higher-level features to be implemented later, but the crbug is https://crbug.com/605350.


Requesting approval to ship?

No. This is just adding low-level support that will be used by higher-level features to be implemented later.

Addy Osmani

unread,
Aug 17, 2016, 5:48:10 PM8/17/16
to blink-dev, be...@chromium.org, ta...@chromium.org, oj...@chromium.org, kenji...@chromium.org, scli...@chromium.org
With respect to "Defer loading of below-the-fold images until the user scrolls down to them", is the short-term goal to just provide this mechanism for the Data Saver use case? You mention higher-level features may use this later but it's unclear what these are. Would it be possible to allude to them?

Deferred image loading has been discussed in a few different forms before and I'm curious if this work is keeping them in mind as a developer facing option for opting into this behavior may be desirable by some.

Scott Little

unread,
Aug 17, 2016, 7:33:24 PM8/17/16
to Addy Osmani, blink-dev, Ben Greenstein, Tal Oppenheimer, Ojan Vafai, Kenji Baheux
The short-term goal is to provide this mechanism for the Data Saver use case. There has been some talk of doing this image replacement for other features, such as perhaps deferring loading of below-the-fold images, but the only concrete use case that I know of right now is the Data Saver one.

Ideally, this mechanism for image replacement using range requests would be usable by any other features that would like to activate it.

Ojan Vafai

unread,
Aug 17, 2016, 8:24:38 PM8/17/16
to Scott Little, Addy Osmani, blink-dev, Ben Greenstein, Tal Oppenheimer, Kenji Baheux
I'd like to eventually experiment with deferring load of below-the-fold images, but that will be a change that needs careful compatibility testing and will require explicit control as well like WICG thread you linked to. I'd like to see both happen eventually, but in the short-term, this will build the infrastructure we'd need for it and only ship it for Data Saver users.

Jeremy Roman

unread,
Aug 17, 2016, 10:13:49 PM8/17/16
to Ojan Vafai, Scott Little, Addy Osmani, blink-dev, Ben Greenstein, Tal Oppenheimer, Kenji Baheux
A couple quick questions about how this is observed by web content (once it's used by Data Saver or some other feature, naturally):

1. Does the load event fire on an <img> when the placeholder loads, or when the original image loads?

2. If a script accesses the image (for example, by drawing it into a CanvasRenderingContext2D), does it see the placeholder image or does it act as though the image has not yet loaded?

Scott Little

unread,
Aug 17, 2016, 10:35:33 PM8/17/16
to Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Ben Greenstein, Tal Oppenheimer, Kenji Baheux
Answers inline.

On Wed, Aug 17, 2016 at 7:13 PM, Jeremy Roman <jbr...@chromium.org> wrote:
A couple quick questions about how this is observed by web content (once it's used by Data Saver or some other feature, naturally):

At least for the Data Saver use case, it should be just as if the server had returned the placeholder image. Other features may want the placeholders to be observed differently by web content, but we can cross that bridge when we come to it.
 

1. Does the load event fire on an <img> when the placeholder loads, or when the original image loads?

The load event would fire when the placeholder loads.
 

2. If a script accesses the image (for example, by drawing it into a CanvasRenderingContext2D), does it see the placeholder image or does it act as though the image has not yet loaded?

The script would see the placeholder image.

PhistucK

unread,
Aug 18, 2016, 1:57:32 AM8/18/16
to Scott Little, Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Ben Greenstein, Tal Oppenheimer, Kenji Baheux
Well, that sounds like a huge compatibility risk, really. Stating "None" is misleading...


PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

wox...@gmail.com

unread,
Aug 18, 2016, 3:52:24 AM8/18/16
to blink-dev, jbr...@chromium.org, oj...@chromium.org, ad...@google.com, be...@chromium.org, ta...@chromium.org, kenji...@chromium.org, scli...@chromium.org
Wouldn't it be more appropriate to add the loadedmetadata event used by HTMLMediaElement? Seems much more obvious and safe to me, the layman.

Stephen Chenney

unread,
Aug 18, 2016, 11:08:39 AM8/18/16
to Scott Little, Ben Greenstein, PhistucK, Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Tal Oppenheimer, Kenji Baheux
Phistuck is absolutely right. Using the placeholder in canvas operations, or any operation triggered by the load event, means broken content unless the page somehow finds out when the actual image has loaded. What is the impact on advertising image display, for example, where it presumably matters to someone that the real image was displayed?

There will also be broken content when an image contains transparent areas intended to reveal content behind the image. There may also be performance implications related to image opacity information being incorrect. The design doc should cover your plans for images with alpha channels.

And the issue of sprite maps has been raised on the doc. Pulling information about sprite maps out of Blink is non-trivial if you also want to avoid loading large element background images.

Cheers,
Stephen.

Scott Little

unread,
Aug 18, 2016, 4:25:24 PM8/18/16
to Stephen Chenney, Ben Greenstein, PhistucK, Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Tal Oppenheimer, Kenji Baheux
I think there might be some misunderstanding here - this intent to implement is just about adding support for showing placeholders, not about deciding which images should be replaced with placeholders.

For the Data Saver use case, the idea is that the user does not want to pay the data cost of downloading the entire image, and the primary reason to show a sized placeholder at all (instead of just eliding images entirely) is to better preserve the page layout. In the majority of cases, the full image will never be loaded, and the user will just see the placeholders. As far as web content is concerned, it will be as if the server returned the placeholder image.

This could of course break some page content (namely the replaced images, since placeholders would be shown in their places). But that would be a compatibility risk of the higher-level features (such as the Data Saver use case) that activate this image replacement support, and not a compatibility risk of this image replacement support itself.

PhistucK

unread,
Aug 18, 2016, 4:48:59 PM8/18/16
to Scott Little, Stephen Chenney, Ben Greenstein, Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Tal Oppenheimer, Kenji Baheux
Why will the Data Saver not implement it at its layer and then Blink will just really get those placeholders from the server (the proxy)?


PhistucK

Scott Little

unread,
Aug 18, 2016, 5:18:40 PM8/18/16
to PhistucK, Stephen Chenney, Ben Greenstein, Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Tal Oppenheimer, Kenji Baheux
On Thu, Aug 18, 2016 at 1:48 PM, PhistucK <phis...@gmail.com> wrote:
Why will the Data Saver not implement it at its layer and then Blink will just really get those placeholders from the server (the proxy)?

The Data Saver proxy cannot transform https:// content, which is roughly half of downloaded bytes. Data Saver does already have a feature called Lo-Fi that generates and returns sized placeholders from the proxy, but that only works for resources that can be proxied through and transformed by the Data Saver proxy.

The image replacement support referred to by this intent to implement would offer a means of showing image placeholders that doesn't require a proxy, and thus would work for https:// resources.

Chris Blume

unread,
Aug 23, 2016, 4:46:18 AM8/23/16
to Scott Little, PhistucK, Stephen Chenney, Ben Greenstein, Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Tal Oppenheimer, Kenji Baheux
I like this for a reason that wasn't mentioned:
- We can raster a tile that doesn't have the image decoded yet. Later, we can swap in the decoded image. Since image decodes are slow, that tile waits an uncomfortably long time.


I am curious about the load event though. Some sites use this to know when an image has been completely downloaded before displaying it, right? Wouldn't that be broken?


Chris Blume |
 Software Engineer | cbl...@google.com | +1-614-929-9221

Scott Little

unread,
Aug 23, 2016, 3:16:01 PM8/23/16
to Chris Blume, PhistucK, Stephen Chenney, Ben Greenstein, Jeremy Roman, Ojan Vafai, Addy Osmani, blink-dev, Tal Oppenheimer, Kenji Baheux
On Tue, Aug 23, 2016 at 1:45 AM, Chris Blume <cbl...@google.com> wrote:
I like this for a reason that wasn't mentioned:
- We can raster a tile that doesn't have the image decoded yet. Later, we can swap in the decoded image. Since image decodes are slow, that tile waits an uncomfortably long time. 
  
 
I am curious about the load event though. Some sites use this to know when an image has been completely downloaded before displaying it, right? Wouldn't that be broken?

See my earlier posts - the short term plan here is to support the Data Saver use case, where the goal is to save data, and the full original image hopefully won't ever need to be downloaded entirely. It'll be as if the server responded with the placeholder, as far as web content is concerned. If other use cases in the future want to trigger load events differently, then we can cross that bridge when we come to it.

This intent to implement is just about adding support for downloading just a few bytes of an image to show a placeholder in its place. It'll be up to the higher level features (such as the Data Saver use case) that activate this image replacement to determine which images to replace and properly balance data savings vs. page usability/brokenness. But that's outside the scope of this intent to implement.

Inian Parameshwaran

unread,
Aug 23, 2016, 8:12:25 PM8/23/16
to blink-dev, be...@chromium.org, ta...@chromium.org, oj...@chromium.org, kenji...@chromium.org, scli...@chromium.org
The image replacement support referred to by this intent to implement would offer a means of showing image placeholders that doesn't require a proxy, and thus would work for https:// resources.

I know that this is not in the scope of this intent, but does that mean enabling the data saver mode would result in images getting replaced in HTTPS pages too? I do not think developers would be very comfortable with that.

Ben Greenstein

unread,
Aug 24, 2016, 2:48:03 PM8/24/16
to Inian Parameshwaran, blink-dev, ta...@chromium.org, oj...@chromium.org, kenji...@chromium.org, scli...@chromium.org
On Tue, Aug 23, 2016 at 5:12 PM Inian Parameshwaran <inia...@gmail.com> wrote:
The image replacement support referred to by this intent to implement would offer a means of showing image placeholders that doesn't require a proxy, and thus would work for https:// resources.

I know that this is not in the scope of this intent, but does that mean enabling the data saver mode would result in images getting replaced in HTTPS pages too? I do not think developers would be very comfortable with that.

Images in both HTTP and HTTPS pages will be replaced under extreme conditions, e.g., when the network is too slow to support downloading the full page. Developers haven't been uncomfortable with this on HTTP pages when image replacement was coordinated through the data reduction proxy. Why would it be different on HTTPS?   

Rick Byers

unread,
Aug 24, 2016, 4:00:30 PM8/24/16
to Ben Greenstein, Inian Parameshwaran, blink-dev, ta...@chromium.org, Ojan Vafai, kenji...@chromium.org, scli...@chromium.org
Would this whole debate about the cost/harm of breaking the page largely go away if instead of thinking of this as a "image replacement" feature we saw it as a "image load suspension" feature?  I.e. after the image header is downloaded and streamed to blink, the network request is suspended indefinitely (as it could be already on a bad network).  Blink would be smart enough to use the header to reserve the proper size but otherwise doesn't consider the image as "loaded" (same as if it was specified in the <img> attributes but the load hadn't started)?

Is it just engineering cost/complexity that's keeping us from taking such a non-web-breaking approach to solving this problem?

PhistucK

unread,
Aug 24, 2016, 4:14:01 PM8/24/16
to Rick Byers, Ben Greenstein, Inian Parameshwaran, blink-dev, Tal Oppenheimer, Ojan Vafai, kenji...@chromium.org, Scott Little
That would break the window load event promise, though, right? It makes more sense to fire the error event if the image is never going to be loaded, I think.


PhistucK

Rick Byers

unread,
Aug 24, 2016, 5:52:48 PM8/24/16
to PhistucK, Ben Greenstein, Inian Parameshwaran, blink-dev, Tal Oppenheimer, Ojan Vafai, kenji...@chromium.org, Scott Little
On Wed, Aug 24, 2016 at 4:13 PM, PhistucK <phis...@gmail.com> wrote:
That would break the window load event promise, though, right? It makes more sense to fire the error event if the image is never going to be loaded, I think.

Yeah that's probably even more web-compatible (although hopefully most pages are already mostly usable well before every single image has finished loading). 

Charles Harrison

unread,
Aug 24, 2016, 5:58:14 PM8/24/16
to Rick Byers, PhistucK, Ben Greenstein, Inian Parameshwaran, blink-dev, Tal Oppenheimer, Ojan Vafai, kenji...@chromium.org, Scott Little
This depends on what the use case for image replacement. I think we will need both to satisfy consumers of this feature.

If the image will never load by design we should fire the error event, otherwise e.g. for deferred loading cases I think it makes sense to simulate a slow load.

Dimitri Glazkov

unread,
Aug 25, 2016, 12:35:39 PM8/25/16
to Charles Harrison, Rick Byers, PhistucK, Ben Greenstein, Inian Parameshwaran, blink-dev, Tal Oppenheimer, Ojan Vafai, kenji...@chromium.org, Scott Little
On Wed, Aug 24, 2016 at 2:58 PM Charles Harrison <cshar...@chromium.org> wrote:
This depends on what the use case for image replacement. I think we will need both to satisfy consumers of this feature.

If the image will never load by design we should fire the error event, otherwise e.g. for deferred loading cases I think it makes sense to simulate a slow load.

On Wed, Aug 24, 2016 at 5:52 PM, Rick Byers <rby...@chromium.org> wrote:
On Wed, Aug 24, 2016 at 4:13 PM, PhistucK <phis...@gmail.com> wrote:
That would break the window load event promise, though, right? It makes more sense to fire the error event if the image is never going to be loaded, I think.

Yeah that's probably even more web-compatible (although hopefully most pages are already mostly usable well before every single image has finished loading). 

My apologies, I am just now catching up. Reading what Scott says, it seems that the Data Saver machinery doesn't intend to load the actual image anyway, which sort of changes the equation. The placeholders aren't placeholders -- they are substitutes.

IOW, when you load a site in this mode, you won't actually see the pictures. (I am guessing there will be an additional UI to force loading pictures later).

In this case, it seems reasonable to just fire a load event when the substitute image is loaded. Yes, it's not the actual image that the client requested from the server, but that's sort of the point of the feature, right?

:DG<

 

PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.


--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Ben Greenstein

unread,
Aug 25, 2016, 12:41:41 PM8/25/16
to Dimitri Glazkov, Charles Harrison, Rick Byers, PhistucK, Inian Parameshwaran, blink-dev, Tal Oppenheimer, Ojan Vafai, kenji...@chromium.org, Scott Little
Yes, that's indeed the point of the feature, and thanks for the clarification! And yes, at least in Chrome we'll have a context menu option to load in images, as well as UI to reload the page with the original images.
Reply all
Reply to author
Forward
0 new messages