Intent to Implement: Lazily load below-the-fold iframes and images

[Scott Little]

Contact emails

scli...@chromium.org, rajen...@chromium.org, be...@chromium.org

Design doc/Spec

Design doc: https://docs.google.com/document/d/1e8ZbVyUwgIkQMvJma3kKUDg8UUkLRRdANStqKuOIvHg/edit

Spec: TBD

Summary

Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them. For frames, third-party iframes that are meant to be shown to the user are deferred, leaving alone frames that are used for analytics or communication according to heuristics. For images, insert placeholders with appropriate dimensions, which it gets by issuing a range request for the first few bytes of the image. When the user scrolls, deferred frames and images that will be visible soon start loading.

Motivation

Reduce data usage, speed up page loads, and reduce memory usage.

From UMA histograms (Navigation.DeferredDocumentLoading.StatesV4) for Android, out of all cross-origin iframes loaded on pages, over 20% were both (a) likely intended to be shown to the user according to heuristics that attempt to filter out analytics/communication iframes (e.g. filter out 0x0, above or left of the screen, etc.) and (b) were never actually visible to the user, i.e. the user never scrolled all the way down to the iframe.

Preliminary benchmarks with basic prototypes of LazyLoad have shown 18% improvement to Speed Index on typical pages on a simulated 3G network. Other benchmarks have shown up to 35% reduction in typical page size with LazyLoad, see analysis.

Risks

Interoperability and Compatibility

Edge: No signals

Firefox: No signals

Safari: No signals

Web developers:

• Multiple JavaScript libraries attempt to use JavaScript to lazy load images, e.g. link

• A thread on WICG about lazy loading images

• Lazy loading is conceptually very similar to the common infinite scroll design

There are some web compatibility risks with LazyLoad, e.g. the document's "load" event could happen before deferred elements are loaded in, so sites that relied on the old ordering might not work correctly unless they disable LazyLoad. Other compatibility risks can be found listed in the design docs, here and here.

Ergonomics

Deferring images will make use of the image replacement feature in Blink.

Some browser features (e.g. Print, Save Page As) that expect all resources to be fully loaded will need to be updated to force any remaining deferred resources to be loaded in before resolving, more details here.

Activation

The current idea is that, once launched, the feature would be enabled by default, with a way to opt-out, although the exact design here is still TBD.

Debuggability

Deferred iframes and images will be marked as such when looking at the "Elements" view in DevTools, so that web developers can differentiate between a slow-loading resource and an element that was specifically deferred by LazyLoad. Exact UI and plumbing TBD.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Android only, at least initially.

LazyLoad as a feature doesn't need to be Android specific, but the plan is to start with Android since the benefits of LazyLoad will likely have the most impact on Android, and this will also help minimize any negative impact to user experience. If LazyLoad is successful on Android, then desktop platforms will be considered as well.

Link to entry on the feature dashboard

ChromeStatus link: TBD

Requesting approval to ship?

No.

[Brenton Simpson]

I'm concerned about how this will affect users with intermittent connectivity.  What if you open an article when you have connectivity, but try to read it when you don't (for instance, while in transit)?  Currently, everything will be cached for your offline perusal, unless the publisher uses lazy load (e.g. Medium).  With this change, large blocks of content below the fold would be missing when offline.

[Yoav Weiss]

Overall, this is something I heard people want for a long while, so thanks for working on this!

With that said, I have some concerns about the approach.

Are there plans to change the spec in order to match that behavior?

* The HTML spec required that images delay the load event of the document they were fetched in. Will that change?

* The image element's load event fires only after its entire data was fetched? Will that change?

* Same questions for iframes.

If the answer to the above is true, are those changes web compatible? Is the plan to experiment with those changes to see their impact in the wild? If so, how are we planning to measure breakage?

Otherwise, the design document mentions fetching using range requests the first 2K of the image in order to get its dimensions, saying it is "highly likely to contain the image dimensions for larger images, according to UMA data (Blink.DecodedImage.EffectiveDimensionsLocation.* histograms)."

Is it possible to share that data publicly? Would be very interesting to know what "highly likely" means.

Also, what is the plan for images that don't exhibit this property?

And for progressive images (read: JPEGs) that fit their initial scan in that range - will that be displayed as a placeholder?

The design document also mentions a dependency on the Data Reduction Proxy - can you elaborate what that dependency is?

For images that are lazy loaded as they are outside of the initial viewport, would it be possible to include the layout based client hints? (as their display dimensions are known at the time their fetch starts)

Will users have a way to opt-out of that behavior, to address Brenton's concerns? Is there an alternative (heuristic) approach that will address them?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d84cdb3a-5114-498b-876c-54cc0b810ef5%40chromium.org.

[Jacob G]

What exactly defines "near them"? 

If the data connection is slow and the user is only two flings/scrolls away from the image, I'd be annoyed: "Why hasn't it loaded already?" - in that case, lazy loading makes reading an article "lazy waiting" (Medium on desktop has this annoyance too).

In general, I think this is a good idea, but there are implications that need to be thought of.

[Stephan Bijzitter]

>Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them.

Could we change this to: 

Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them or until all content above the fold has finished loading.

That'd solve the problem for opening an article and reading it while offline, and it would also improve the experience for fast scrolling people. 

[Patrick Meenan]

Is the plan to enable this for all users or just when data savings is enabled?  I'd be supportive of doing it in data saver mode but have some pretty big concerns about doing it across the board.

Fetching the images after the network has gone idle can have a lot of latency, particularly if the cell connection needs to ramp back up (not to mention using a lot of power going between states).  There's a pretty high likelihood that users will be scrolling to blank content that needs to fill in and it's one of the bigger problems with js-based lazy-loaders (my preference when working with them is to delay the images until after the main content loads but to still load all of the images and not wait for user interaction).

It looks like the range-request mechanism is only used for cases where dimensions aren't known.  Any idea what the risk is of breaking tracking and retargeting pixels?

[Scott Little]

Replies inline.

On Wed, Jan 31, 2018 at 5:53 AM, 'Patrick Meenan' via blink-dev <blin...@chromium.org> wrote:

Is the plan to enable this for all users or just when data savings is enabled?  I'd be supportive of doing it in data saver mode but have some pretty big concerns about doing it across the board.

Currently, we're mainly focused on getting a dev channel experiment going and seeing how it works in the wild. Ideally LazyLoad would be advantageous enough for it to be turned on across the board, but that depends on how the experiments go.

 

Fetching the images after the network has gone idle can have a lot of latency, particularly if the cell connection needs to ramp back up (not to mention using a lot of power going between states).  There's a pretty high likelihood that users will be scrolling to blank content that needs to fill in and it's one of the bigger problems with js-based lazy-loaders (my preference when working with them is to delay the images until after the main content loads but to still load all of the images and not wait for user interaction).

See the doc. TL;DR the idea is to tune the load-in distance threshold so that stuff starts loading a bit *before* the deferred content is visible, such that deferred content is almost always fully loaded in by the time the user has scrolled to it.

 

It looks like the range-request mechanism is only used for cases where dimensions aren't known.  Any idea what the risk is of breaking tracking and retargeting pixels?

See the relevant section in the doc.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/4c151882-0a67-4bc0-9306-14db69f3a364%40chromium.org.

[Scott Little]

Replies inline.

On Wed, Jan 31, 2018 at 1:25 AM, Stephan Bijzitter <stephanb...@gmail.com> wrote:

>Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them.
Could we change this to: 

Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them or until all content above the fold has finished loading.

Unfortunately, that would mean the user loses out on data savings and memory usage savings if they never end up scrolling to the content.

 

That'd solve the problem for opening an article and reading it while offline, and it would also improve the experience for fast scrolling people. 

See the doc. TL;DR the idea is to tune the load-in distance threshold so that stuff starts loading a bit *before* the deferred content is visible, such that deferred content is almost always fully loaded in by the time the user has scrolled to it.

The possible issues with offline reading are also on our radar, see the doc.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a57a2a40-fa36-469a-9cd1-a3e200c5c4e9%40chromium.org.

[Scott Little]

On Tue, Jan 30, 2018 at 11:48 PM, Jacob G <kurte...@gmail.com> wrote:

What exactly defines "near them"? 

If the data connection is slow and the user is only two flings/scrolls away from the image, I'd be annoyed: "Why hasn't it loaded already?" - in that case, lazy loading makes reading an article "lazy waiting" (Medium on desktop has this annoyance too).

In general, I think this is a good idea, but there are implications that need to be thought of.

See the doc for an explanation of the different factors that will be taken into consideration for determining the load-in distance threshold (link) and a plan for how we can experimentally tune these distance thresholds to minimize any waiting (link).

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/e6f12bd8-ce8b-4932-bdd4-9707d24febd7%40chromium.org.

[Stephan Bijzitter]

>Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them.
Could we change this to: 

Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them or until all content above the fold has finished loading.

Unfortunately, that would mean the user loses out on data savings and memory usage savings if they never end up scrolling to the content.

While that's absolutely true, it would still be an improvement over the current behaviour and perhaps with powersaving mode (or simply a setting) it could be made more passive.
I just fear this feature will be useful for the majority of people, while being quite a nuisance for power users like myself.

Scrolling on a macbook's touchpad is incredibly fast and using the scrollbar is instantaneous. If I open a website, read the content above the fold and start scrolling after one minute, I would expect the entire website to be fully loaded but with this feature implemented as its currently planned, it would still need to load more content even after a minute. Now that's visually annoying, but if the page starts jumping around because of images filling in the gaps... ouch.

Perhaps it would be more prudent to implement this feature and have it disabled by default for a while, allowing developers to select one out of several modes and try them out to see what it would do for their own experience, and for the experience of their customers. After feedback has been collected, a new default value could be chosen and developers will have already had some time to adjust their websites to make optimal usage for this feature.

[Scott Little]

On Tue, Jan 30, 2018 at 10:39 PM, 'Brenton Simpson' via blink-dev <blin...@chromium.org> wrote:

I'm concerned about how this will affect users with intermittent connectivity.  What if you open an article when you have connectivity, but try to read it when you don't (for instance, while in transit)?  Currently, everything will be cached for your offline perusal, unless the publisher uses lazy load (e.g. Medium).  With this change, large blocks of content below the fold would be missing when offline.

It's true that there are some situations where this could annoy users, mainly if there are deferred images further down the article page that the user both wanted to see and expected to be fully loaded, but at least the text and other parts of the article are intact. Infinite scrolling is common enough that users will at least hopefully be used to this kind of stuff-loads-in-when-it's-scrolled-to behavior, plus I'm not sure how common this kind of open-earlier and read-offline-later use case is.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d84cdb3a-5114-498b-876c-54cc0b810ef5%40chromium.org.

[Scott Little]

On Tue, Jan 30, 2018 at 11:39 PM, Yoav Weiss <yo...@yoav.ws> wrote:

Overall, this is something I heard people want for a long while, so thanks for working on this!

With that said, I have some concerns about the approach.

Are there plans to change the spec in order to match that behavior?

* The HTML spec required that images delay the load event of the document they were fetched in. Will that change?

Yep, the spec there might have to be to be modified depending on how the final design looks. Exact changes TBD.

 

* The image element's load event fires only after its entire data was fetched? Will that change?

The current design is to delay the image element's load event until the full image is loaded in. See the doc.

 

* Same questions for iframes.

Same answer as above for images :)

 

If the answer to the above is true, are those changes web compatible? Is the plan to experiment with those changes to see their impact in the wild? If so, how are we planning to measure breakage?

The plan is to experiment to see the impact in the wild. Unfortunately, there's not a lot of ways to measure how much or how badly pages are broken by this, especially since (at least for iframes) most of the ways this could break pages might not be very user visible.

The current idea is that we'll do a bunch of manual testing on popular websites (on top of the manual testing on popular websites that has already been done, to figure out e.g. the heuristics for detecting communications/analytics iframes), as well as gradually roll out the feature with lots of blogposts and other communications to hopefully avoid catching anyone off guard.

 

Otherwise, the design document mentions fetching using range requests the first 2K of the image in order to get its dimensions, saying it is "highly likely to contain the image dimensions for larger images, according to UMA data (Blink.DecodedImage.EffectiveDimensionsLocation.* histograms)."

Is it possible to share that data publicly? Would be very interesting to know what "highly likely" means.

These histograms have since been deprecated, but "highly likely" here means >99% for GIF, >97% for JPEG, and >96% for PNG.

 

Also, what is the plan for images that don't exhibit this property?

And for progressive images (read: JPEGs) that fit their initial scan in that range - will that be displayed as a placeholder?

Right now, there's no special handling for progressive JPEGs, although that could be added to the image replacement feature if it turns out to be worth it, see that doc.

 

The design document also mentions a dependency on the Data Reduction Proxy - can you elaborate what that dependency is?

The only interaction with DRP stuff is with the client code that exposes data savings to the user, i.e. we want to show data savings from LazyLoad together with data savings due to DRP, Previews, etc. if the user has Data Saver turned on.

There's no special DRP server dependencies or anything like that, at least not in the current design.

 

For images that are lazy loaded as they are outside of the initial viewport, would it be possible to include the layout based client hints? (as their display dimensions are known at the time their fetch starts)

That's definitely possible, e.g. we could even just avoid doing a range request at all for images that have the width and height specified in the element itself, but I'm worried that this would break stuff like tracking pixels at the bottom of a page, causing these tracking pixels to never get fetched. Issuing a range request still allows the web server to record that the tracking pixel was fetched.

 

Will users have a way to opt-out of that behavior, to address Brenton's concerns? Is there an alternative (heuristic) approach that will address them?

Right now, there isn't any user opt-out planned. The idea is that web pages will be able to opt-out, but that doesn't necessarily solve the issue Brenton brought up. See my earlier response to his concern on this thread.

 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d84cdb3a-5114-498b-876c-54cc0b810ef5%40chromium.org.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEhjDPLKSLMpjKk0duP3Gea7S3cm3xosdAVGiBC_DH-i2Q%40mail.gmail.com.

[Scott Little]

On Wed, Jan 31, 2018 at 12:21 PM, Stephan Bijzitter <stephanb...@gmail.com> wrote:

>Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them.
Could we change this to: 

Defer loading certain below the fold third-party iframes and below-the-fold images on the page until the user scrolls near them or until all content above the fold has finished loading.

Unfortunately, that would mean the user loses out on data savings and memory usage savings if they never end up scrolling to the content.

While that's absolutely true, it would still be an improvement over the current behaviour and perhaps with powersaving mode (or simply a setting) it could be made more passive.
I just fear this feature will be useful for the majority of people, while being quite a nuisance for power users like myself.

Scrolling on a macbook's touchpad is incredibly fast and using the scrollbar is instantaneous.

FYI, we're initially planning to launch this for Android only, at least to start off; see the doc and the Intent to Implement at the start of this thread.

 

If I open a website, read the content above the fold and start scrolling after one minute, I would expect the entire website to be fully loaded but with this feature implemented as its currently planned, it would still need to load more content even after a minute.

The plan is to tune the load-in distance-from-viewport threshold such that users almost never need to wait for content. See the description of it in the doc and also the plan to tune the distance threshold experimentally in the same doc.

 

Now that's visually annoying, but if the page starts jumping around because of images filling in the gaps... ouch.

FYI, the image replacement mechanism specifically preserves the page layout by inserting a placeholder with the same dimensions as the original image, so pages shouldn't jump around as stuff loads in.

 

Perhaps it would be more prudent to implement this feature and have it disabled by default for a while, allowing developers to select one out of several modes and try them out to see what it would do for their own experience, and for the experience of their customers. After feedback has been collected, a new default value could be chosen and developers will have already had some time to adjust their websites to make optimal usage for this feature.

Right now, we're mainly focusing on getting a Dev channel experiment going and then proceed from there according to what we learn from the experiment.

That said, we'll definitely want to do a gradual rollout of this feature with lots of communication/blogposts and the like, to avoid catching any web developers off guard and give them time to see how their sites work with this feature in the (hopefully very unlikely) event that they need to adjust their site to work with this feature.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/f7162c9d-1fe2-4293-8b73-f8f8806e9bfd%40chromium.org.

[tom....@spektrix.com]

In principle I'm very behind this but do have concerns with regards to iframe sizing. 

In situations where it's not possible to know the necessary dimensions of an iframe before loading and a scrollbar isn't wanted, we have to use scripts to communicate the size of an iframe to it's parent to then resize the iframe. If the iframe was below the fold and lazy loaded, you wouldn't be able to communicate the size of the iframe and thus resize it until it was nearly displayed. If someone scrolls fast enough and missed the nearly, it's very likely they would then witness the iframe resizing itself making the page look janky. 

Having the "lazyload=off" attribute on the parent's will obviously solve this but as this hasn't been mentioned yet I thought it worth flagging.

[srdjan.p...@gmail.com]

This sounds great for the end user. However, from a developer perspective, I would expect to see this implemented as an opt-in attribute , like <img ... lazy />, <iframe ... lazy />.

Because image dimensions become available only after the image was fully loaded, some preload strategies that wait for all the images to load will fail under the opt-in behaviour. For instance, there is a popular jQuery plugin called imagesLoaded with 7000 stars on GitHub: https://github.com/desandro/imagesloaded. It's primary use was scheduling a callback after images within a container have all loaded. Depending on the placement of those images and the purpose of the callback, the callback might never execute, or execute only once the user has scrolled to the bottom of the page.

I don't know how the cost of potential backwards incompatibility compares to potential bandwidth savings and performance improvement. This feature can greatly improve performance of websites with a large number of external embeds, such as YouTube videos or Facebook buttons. However, the cost of incompatibility can't be easily predicted, and should be thoroughly measured and quantified during the testing period.

Good luck, hope to see this shipped soon, opt-in or out.

Cheers

[Jake Archibald]

Range requests are a source of security issues since they're pretty undefined. I'm working to fix this for media elements, and the spec for lazy loading should do the same. I'll try to abstract as many of the algorithms as possible.

Here are the attacks we should avoid https://github.com/whatwg/fetch/issues/144#issuecomment-368040980.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACBp5rXUQ-CtqTTNpYFMMNWiiTKRMHSWEtdCL%3D8Pv5QGPuvgTg%40mail.gmail.com.

[Scott Little]

On Thu, Apr 19, 2018 at 3:37 PM, Jake Archibald <jakear...@google.com> wrote:

Range requests are a source of security issues since they're pretty undefined. I'm working to fix this for media elements, and the spec for lazy loading should do the same. I'll try to abstract as many of the algorithms as possible.

Here are the attacks we should avoid https://github.com/whatwg/fetch/issues/144#issuecomment-368040980.

I don't think these attacks apply here. When an image gets deferred by lazy load, it fetches the first 2KB of the image with a range request to try to decode the image dimensions and insert a properly sized placeholder. Then later, if the user scrolls down near the image, Blink will make a full request (without any range header) for the full image, so I don't think these attacks that rely on tricking Blink into fetching a particular range of a resource in that second fetch would apply.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Jake Archibald]

True facts. Those attacks rely on joining partials together into one resource, so no worries here. Sorry for the alarm.

The range request should still be specified. https://github.com/whatwg/fetch/pull/560 makes sure they survive going through a service worker.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Anne van Kesteren]

On Fri, Apr 20, 2018 at 2:13 AM, 'Jake Archibald' via blink-dev
<blin...@chromium.org> wrote:
> The range request should still be specified.
> https://github.com/whatwg/fetch/pull/560 makes sure they survive going
> through a service worker.

Indeed, and it should be defined what happens if it fails or if the
server returns the full image. This should be defined as part of
HTML's image loading (which hopefully at some point will be
generalized enough so all the other bits of the platform that load
images also get to use it).


--
https://annevankesteren.nl/