Opinions on img srcset?

[Nico Weber]

Hi,

WebKit recently implemented support for <img srcset>, as described on https://www.webkit.org/blog/2910/improved-support-for-high-resolution-displays-with-the-srcset-image-attribute/ . Does anyone have an opinion on if we should do this too?

Nico

[Ilya Grigorik]

Short answer: yes, we should! But the short answer warrants a longer one too... and it just so happens that I was drafting an email on this exact topic.

---

WebKit landed srcset support earlier this month. srcset is not perfect, and it may not address every single wishlist item from the RICG group, but it's good enough - it addresses the critical use cases, it's "preloader friendly", and it does not preclude other solutions that we can continue to explore and add later (i.e. progressive image formats, pixel perfect delivery, and so on). 

If there is one consistent complaint against srcset, then it's the verbose and cryptic syntax. E.g:

<img src="pear-mobile.jpeg" srcset="pear-mobile-2x.jpeg 640w 2x, pear-mobile-1.5.jpeg 640w 1.5x,  pear-mobile-3x.jpeg 640w 3x, pear-tablet.jpeg 1280w 2x, ...,  ..., pear-desktop.jpeg">

The combination of DPR switching and resolution switching creates an explosion of cases that need to be covered in the markup (# of DPR resolutions X # of resolutions). However, that doesn't need to be the case -- repeating the DPR switching is pure boilerplate, which can and should be automated. If we land Client-Hints (for DPR negotiation) in Chrome, then this problem goes away. (We even have an existing CL, albeit one which needs to be updated).

---

The combination of srcset + CH will give us:

<img src="pear.jpeg">

• Image type is negotiated at the HTTP layer (via Accept)

• DPR selection is negotiated at the HTTP layer (via CH)

• … Same img tag, but HiDPI and multi-format friendly (and easily automated).

  
  

<img src="pear-mobile.jpeg"          srcset="pear-mobile.jpeg 640w, pear-tablet.jpeg 1280w”>

• Same as first example, DPR and image type negotiated at HTTP layer

• srscet provides optional resolution selection + basic art direction

• Backwards compatible with older clients, no new tags, and easy to polyfill

Don't want to use CH? Use srcset syntax... feast your eyes:

<img srscet=" img.jpg 3x, img.jpg 2.5x, img.jpg 2.2x, img.jpg 2x, img.jpg 1.8x, img.jpg 1.7x, img.jpg 1.6x, img.jpg 1.5x, img.jpg 1.3, img.jpg 1x” /> </facepalm>

---

Note that current webkit implementation only implements DPR switching, which is unfortunate.. My actual vote would be to hide the DPR negotiation entirely within HTTP layer -- this functionality can be easily provided by CDN's / image hosting services / or a basic .htaccess file on your Apache server, instead of forcing the site author to modify every image tag. But the feedback that I've received so far is that there should be a markup solution also - srcset provides that.

ig 

[Yoav Weiss]

I agree with Ilya's short answer, but the longer answer warrants a long reply :)

I think we need to distinguish between 3 major use-cases for responsive images:

* DPR based resolution-switching

* Viewport based res-switching

* MQ based art-direction

The latter two may look similar (both can usually be controlled based on viewport width, at least in most cases), but they are inherently different.
With res-switching, the browser can and should fall back to a lower res image if things get rough (low BW, low battery, user setting because of roaming plan, etc). The user in this case, will receive the same image authors intended him to receive, but in lower resolution.

With art-direction, doing so will present the user with a broken layout, or an indecipherable image, resulting in bad UX.

I believe that the two cases of res-switching can and should be merged. That can be done by defining some base viewport width, with the 'x' qualifier redefined as a multiple of the DPR and the base viewport width. (e.g. 3x meaning either one of: "This is 1x DPR with a viewport width of 3*base", "This is 1.5x DPr with a viewport width of 2*base", etc).

Obviously, since this is a change of semantics without changing the syntax, this should happen quickly and would require consensus from the WebKit folks, in order to avoid compatibility issues.

Assuming this makes sense and accepted for srcset, the same "x" semantics can also be applied to CH, resulting in better server side resolution switching, without compromising caching with multiple CH values.

Then, CH and srcset can be exact mirrors of client-side and server-side resolution negotiation, allowing developers pick the solution type that best fits them.

As for the art-direction use-case, I believe it is best kept out of srcset, at least for now. We'll see in the future if it's best addressed by adopting picture or by extending srcset further.

As a final note, I'd like to say that there's a Responsive Images meet-up happening in Paris in 2 weeks time. Several browser vendors are participating (including Google folks). If anyone working on Blink would like to participate (either in person or remotely), feel free to contact me.
After the meetup (and its conclusions), I'd be happy to port the srcset code from WebKit into Blink, and help evolving it.

Yoav

P.S. The "srcset is good enough" doc is great (even if I don't 100% agree with all of its conclusions), but as a nitpick, I'd like to point out that the statement “It’s significant that you cannot qualify the viewport’s size in terms of other units, like ems or inches, ... This is to avoid creating any obstacles for the image preloader, which will go about its business long before the CSS cascade and layout have run to completion." quoted in the doc is no longer true for MQs. The PreloadScanner currently does full MQ evaluation for external stylesheets. The same can be done for images, without waiting for CSS or layout. The statement is true though, regarding image display dimensions based solution.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[igri...@google.com]

I think we need to distinguish between 3 major use-cases for responsive images:

* DPR based resolution-switching

* Viewport based res-switching

* MQ based art-direction

The latter two may look similar (both can usually be controlled based on viewport width, at least in most cases), but they are inherently different.

Yup. The "usually" part is why it's say it's "good enough".. assuming we agree that such mechanism is needed in the first place. Do you know why WebKit skipped the viewport res-switching in their current implementation, and if there are plans to add that?

 

I believe that the two cases of res-switching can and should be merged. That can be done by defining some base viewport width, with the 'x' qualifier redefined as a multiple of the DPR and the base viewport width. (e.g. 3x meaning either one of: "This is 1x DPR with a viewport width of 3*base", "This is 1.5x DPr with a viewport width of 2*base", etc). Obviously, since this is a change of semantics without changing the syntax, this should happen quickly and would require consensus from the WebKit folks, in order to avoid compatibility issues. 

Assuming this makes sense and accepted for srcset, the same "x" semantics can also be applied to CH, resulting in better server side resolution switching, without compromising caching with multiple CH values.

This is new (that's one red flag). Further, this seems counter-productive: you're fabricating a new abstract (and non-intuitive) unit and hiding information in the process. As you pointed out DPR and device res-switching are different problems - no reason to mix them together here. Also, there is a *huge* variety of screen sizes on Android devices (only a few on iOS) -- my Google Analytics shows 129 different resolutions! Multiply that out by number of DPR values, and we're looking at 1000+ variants with a completely abstract unit space "use this image for 1450x" - what's 1450X? Is that an in image for a large screen, or a high DPI one? ... =/

I don't want to side-track the conversation here.. let's keep focused on srcset as it exists today (it's good enough :)) -- it's taken us a long time to get here. We need to break the backlog. 

ig

[Yoav Weiss]

On Tue, Aug 27, 2013 at 11:52 PM, <igri...@google.com> wrote:

I think we need to distinguish between 3 major use-cases for responsive images:

* DPR based resolution-switching

* Viewport based res-switching

* MQ based art-direction

The latter two may look similar (both can usually be controlled based on viewport width, at least in most cases), but they are inherently different.

Yup. The "usually" part is why it's say it's "good enough".. assuming we agree that such mechanism is needed in the first place. Do you know why WebKit skipped the viewport res-switching in their current implementation, and if there are plans to add that?

The reasoning I know is in the initial bug. I'm not aware of any plans to add them, but will discuss it (and the w/h syntax in general) with Ted in the Paris meet-up.

The reason I don't think the w/h syntax is good enough is that it cannot address both viewport res-switching and art direction at the same time. You have to choose either one or the other, when both are extremely important (one for performance optimization and the other for content optimization).

Other than that, mixing both viewport and DPR res-switching can lead to duplicate mentions of the same resource.

E.g.

     <img src="c...@1x.png" srcset="c...@2x.png 2x 320w, c...@2x.png 1x 640w, c...@4x.png 2x 640w, c...@4x.png 1x 1280w">

 

I believe that the two cases of res-switching can and should be merged. That can be done by defining some base viewport width, with the 'x' qualifier redefined as a multiple of the DPR and the base viewport width. (e.g. 3x meaning either one of: "This is 1x DPR with a viewport width of 3*base", "This is 1.5x DPr with a viewport width of 2*base", etc). Obviously, since this is a change of semantics without changing the syntax, this should happen quickly and would require consensus from the WebKit folks, in order to avoid compatibility issues. 

Assuming this makes sense and accepted for srcset, the same "x" semantics can also be applied to CH, resulting in better server side resolution switching, without compromising caching with multiple CH values.

This is new (that's one red flag).

Maybe an orange flag? :) I agree that introducing new stuff at this stage is late, but it's not yet too late IMO. Maybe I'm wrong.

 

Further, this seems counter-productive: you're fabricating a new abstract (and non-intuitive) unit and hiding information in the process. As you pointed out DPR and device res-switching are different problems - no reason to mix them together here. Also, there is a *huge* variety of screen sizes on Android devices (only a few on iOS) -- my Google Analytics shows 129 different resolutions! Multiply that out by number of DPR values, and we're looking at 1000+ variants with a completely abstract unit space "use this image for 1450x" - what's 1450X? Is that an in image for a large screen, or a high DPI one? ... =/

 

The fabricated, abstract unit will reduce the number of permutations, not increase it. Let's not call it "x" then, but replace it with an 'm' (for multiple), where 1m means a "1x 320w" equivalent.

For the example above that would be:
    <img src="c...@1x.png" srcset="c...@2x.png 2m, c...@4x.png 4m">

I also think that CH (with its constraint on sending a single value, to avoid caching issues) will benefits from such a unit that on the one hand doesn't expose all the information on DPR and resolution (so reduces the fingerprinting risk that raises resistance to the feature) and on the other hand will give the server exactly the information it needs in order to send the correct resolution image.

Also, when doing res-switching, you don't have to enumerate *all* the possible resolutions that you'd come across, only the ones where the produced image sizes are far apart enough. (And a build step can take care of that, even for users that can't control their server config and that cannot change their hosting provider)

if we aim to resolve all issues with the srcset syntax, since we have 3 problems and 2 syntactic tools to address them, unless you're mixing together problems, you have to leave out one of them. I prefer mixing together both resolution switching problems (especially if it results in DRYer syntax), than to mix together viewport res-switching and art-direction, which are inherently different.

[Tab Atkins Jr.]

On Tue, Aug 27, 2013 at 12:33 AM, Yoav Weiss <yo...@yoav.ws> wrote:

I agree with Ilya's short answer, but the longer answer warrants a long reply :)

I think we need to distinguish between 3 major use-cases for responsive images:

* DPR based resolution-switching

* Viewport based res-switching

* MQ based art-direction

The latter two may look similar (both can usually be controlled based on viewport width, at least in most cases), but they are inherently different.
With res-switching, the browser can and should fall back to a lower res image if things get rough (low BW, low battery, user setting because of roaming plan, etc). The user in this case, will receive the same image authors intended him to receive, but in lower resolution.

With art-direction, doing so will present the user with a broken layout, or an indecipherable image, resulting in bad UX.

What is "viewport based res-switching"?  In what cases would you do that, and to what effect?

~TJ

[Ilya Grigorik]

I think we need to distinguish between 3 major use-cases for responsive images:

* DPR based resolution-switching

* Viewport based res-switching

* MQ based art-direction

The latter two may look similar (both can usually be controlled based on viewport width, at least in most cases), but they are inherently different.

Yup. The "usually" part is why it's say it's "good enough".. assuming we agree that such mechanism is needed in the first place. Do you know why WebKit skipped the viewport res-switching in their current implementation, and if there are plans to add that?

The reasoning I know is in the initial bug. I'm not aware of any plans to add them, but will discuss it (and the w/h syntax in general) with Ted in the Paris meet-up.

Got it, thanks. Agreed, this sounds like a great topic to discuss at the meetup in Paris. 

On a somewhat related note, do you know what the status of srcset on Mozilla/IE side?

 

The reason I don't think the w/h syntax is good enough is that it cannot address both viewport res-switching and art direction at the same time. You have to choose either one or the other, when both are extremely important (one for performance optimization and the other for content optimization).

Other than that, mixing both viewport and DPR res-switching can lead to duplicate mentions of the same resource.

E.g.

     <img src="c...@1x.png" srcset="c...@2x.png 2x 320w, c...@2x.png 1x 640w, c...@4x.png 2x 640w, c...@4x.png 1x 1280w">

Right, hence my motivation for pushing DPR switching into HTTP layer -- the above example only shows the 2x case, so the problem is actually much worse.

 

I believe that the two cases of res-switching can and should be merged. That can be done by defining some base viewport width, with the 'x' qualifier redefined as a multiple of the DPR and the base viewport width. (e.g. 3x meaning either one of: "This is 1x DPR with a viewport width of 3*base", "This is 1.5x DPr with a viewport width of 2*base", etc). Obviously, since this is a change of semantics without changing the syntax, this should happen quickly and would require consensus from the WebKit folks, in order to avoid compatibility issues. 

Assuming this makes sense and accepted for srcset, the same "x" semantics can also be applied to CH, resulting in better server side resolution switching, without compromising caching with multiple CH values.

This is new (that's one red flag).

Maybe an orange flag? :) I agree that introducing new stuff at this stage is late, but it's not yet too late IMO. Maybe I'm wrong.

I think this discussion is better continued in a separate thread and as a separate initiative. *If* it pans out, that's a major revision to the spec (effectively, a v2), at which point the implementations can be updated -- optimistically, if that were to happen, that would still take a while. (Personally, I'm not on the optimistic side based on description so far).

In the meantime, we know that DPR-switching is a *big problem*, and we need a solution. srcset with DPR switching addresses that, and we should move forward - we've delayed long enough.

ig

[Ilya Grigorik]

On Wed, Aug 28, 2013 at 11:18 AM, Tab Atkins Jr. <jacka...@gmail.com> wrote:

What is "viewport based res-switching"?  In what cases would you do that, and to what effect?

Based on the RICG definitions:

- http://usecases.responsiveimages.org/#resolution-switching

- http://usecases.responsiveimages.org/#art-direction

 

[Ilya Grigorik]

On Wed, Aug 28, 2013 at 2:34 AM, Addy Osmani <ad...@google.com> wrote:

Ilya, do you know if Ben Greenstein still has plans to update his existing CL (which you mentioned in the thread)? I ask as it may be useful to get Yoav's help bringing it up to date otherwise.

Just talked with Ben.. Yep, he's game to update it / simplify it to focus on DPR switching.

[Robert Flack]

Seems like other people have touched on most of the issues. However even if srcset was otherwise perfect, many images are specified in CSS properties. I don't have the statistics handy, but custom borders, backgrounds, button styles, scrollbars, custom bullets, etc are specified in and belong in CSS but also should support DPR switching. I worry about trying to invent the perfect solution just for img/picture tags and having to resort to what we've deemed sub-optimal approaches in stylesheets.

Or could the srcset syntax eventually make its way to CSS image selection?

[Boris Smus]

Robert, there is already an equivalent CSS feature called image-set. It's available now as -webkit-image-set.

[Robert Flack]

I wouldn't say image-set is equivalent to srcset. It provides an image per scale but doesn't touch on things like designer breakpoints and has yet another syntax for specifying each.

[Robert Flack]

Sorry, hit send to early. With that being said, I fully support bringing <img> tags up to parity with CSS image-set. I'm just concerned with having potentially diverging specs on each and would prefer some unified spec.

Currently we have <img srcset="foo.jpg 1x, f...@2x.jpg 2x"> vs

<img style="content: image-set(url('foo.jpg') 1x, url('f...@2x.jpg') 2x);"> (or having the CSS in its own file of course).

I assume the rules regarding when to switch between 1x and 2x will be the same?

[Boris Smus]

You're right -- I was also too hasty in suggesting that @srcset and image-set are the same. However, the fact that they are different in subtle ways is a bad thing.

I think the difference in behavior and syntax of the two is very confusing. As a first step, I would love to see both of them converge to the same behavior, and for them to both support the density case (1x, 2x, etc).

The viewport-based w/h breakpoints are very complex. As specified, I think that the behavior is difficult to intuit and internalize (see the many step algorithm in the spec), so many designers and developers will make mistakes.

[Yoav Weiss]

On Wed, Aug 28, 2013 at 10:25 PM, Ilya Grigorik <igri...@google.com> wrote:

I think we need to distinguish between 3 major use-cases for responsive images:

* DPR based resolution-switching

* Viewport based res-switching

* MQ based art-direction

The latter two may look similar (both can usually be controlled based on viewport width, at least in most cases), but they are inherently different.

Yup. The "usually" part is why it's say it's "good enough".. assuming we agree that such mechanism is needed in the first place. Do you know why WebKit skipped the viewport res-switching in their current implementation, and if there are plans to add that?

The reasoning I know is in the initial bug. I'm not aware of any plans to add them, but will discuss it (and the w/h syntax in general) with Ted in the Paris meet-up.

Got it, thanks. Agreed, this sounds like a great topic to discuss at the meetup in Paris. 

On a somewhat related note, do you know what the status of srcset on Mozilla/IE side?

 

Mozilla have a bug open, but no official answer besides "we need to add this attribute to MDN"

I've got nothing but a "no comment" from the IE guys in the past, but since they should show up in Paris, I'll try to bug them in person there.
 

 

The reason I don't think the w/h syntax is good enough is that it cannot address both viewport res-switching and art direction at the same time. You have to choose either one or the other, when both are extremely important (one for performance optimization and the other for content optimization).

Other than that, mixing both viewport and DPR res-switching can lead to duplicate mentions of the same resource.

E.g.

     <img src="c...@1x.png" srcset="c...@2x.png 2x 320w, c...@2x.png 1x 640w, c...@4x.png 2x 640w, c...@4x.png 1x 1280w">

Right, hence my motivation for pushing DPR switching into HTTP layer -- the above example only shows the 2x case, so the problem is actually much worse.

 

I believe that the two cases of res-switching can and should be merged. That can be done by defining some base viewport width, with the 'x' qualifier redefined as a multiple of the DPR and the base viewport width. (e.g. 3x meaning either one of: "This is 1x DPR with a viewport width of 3*base", "This is 1.5x DPr with a viewport width of 2*base", etc). Obviously, since this is a change of semantics without changing the syntax, this should happen quickly and would require consensus from the WebKit folks, in order to avoid compatibility issues. 

Assuming this makes sense and accepted for srcset, the same "x" semantics can also be applied to CH, resulting in better server side resolution switching, without compromising caching with multiple CH values.

This is new (that's one red flag).

Maybe an orange flag? :) I agree that introducing new stuff at this stage is late, but it's not yet too late IMO. Maybe I'm wrong.

I think this discussion is better continued in a separate thread and as a separate initiative. *If* it pans out, that's a major revision to the spec (effectively, a v2), at which point the implementations can be updated -- optimistically, if that were to happen, that would still take a while. (Personally, I'm not on the optimistic side based on description so far).

In the meantime, we know that DPR-switching is a *big problem*, and we need a solution. srcset with DPR switching addresses that, and we should move forward - we've delayed long enough.

 

Sure, as long as it's clear we're solving only a part of the problem, and we'll need to further iterate on the solution until the major use cases are solved.

[Ilya Grigorik]

On Thu, Aug 29, 2013 at 12:45 AM, Yoav Weiss <yo...@yoav.ws> wrote:

In the meantime, we know that DPR-switching is a *big problem*, and we need a solution. srcset with DPR switching addresses that, and we should move forward - we've delayed long enough.

 

Sure, as long as it's clear we're solving only a part of the problem, and we'll need to further iterate on the solution until the major use cases are solved.

Can't +1 this hard enough. Amen.

[Tab Atkins Jr.]

On Wed, Aug 28, 2013 at 4:47 PM, Robert Flack <fla...@chromium.org> wrote:
> I wouldn't say image-set is equivalent to srcset. It provides an image per
> scale but doesn't touch on things like designer breakpoints and has yet
> another syntax for specifying each.

Viewport-based stuff, designer breakpoints, and the like are all
handled by Media Queries in CSS. image-set() covers the resolution
case, which isn't yet addressed in MQs and shouldn't ever be
(http://www.xanthir.com/blog/b4Hv0).

srcset's syntax attempts to marry the two by inventing a new syntax
for a limited subset of MQs.

~TJ

[Christian Biesinger]

I'm interested in working on a blink implementation here, since it
sounds like nobody else volunteered so far :)

It is unfortunate that despite the similar name, the attribute and the
CSS property support different features... but I guess that's
unavoidable, since CSS media queries don't let you change an img's
src, and expanding CSS image-set to include stuff that's redundant
with media queries is also unlikely.

-christian

[Rik Cabanier]

Now that blink is going to tie devicePixelRatio to browser zoom, will this complicate the implementation of srcset?

Or should browser zoom not affect what is chosen with srcset? (according to the spec it should)

WebKit doesn't change DPR and Firefox doesn't implement srcset, so blink would be the first one who has to deal with this.

On Mon, Aug 26, 2013 at 3:11 PM, Nico Weber <tha...@chromium.org> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Yoav Weiss]

On Wed, Sep 4, 2013 at 7:24 AM, Rik Cabanier <caba...@gmail.com> wrote:

Now that blink is going to tie devicePixelRatio to browser zoom, will this complicate the implementation of srcset?

Or should browser zoom not affect what is chosen with srcset? (according to the spec it should)

 

I think we have 3 cases here:

* First time the resource is requested -Here the resource chosen should be impacted by DPR. As far as I understand it, it should already work with the current patch, since deviceScaleFactor should already return the zoom impacted value. Haven't tested it yet, so if I'm wrong, please let me know.

* User zoomed out, decreasing the DPR - If a high-quality resource was already downloaded, no need to re-download a lower quality resource.

* User zoomed in, increasing the DPR - Here we should re-download the resource, but not unconditionally. This will need some heuristics in place that impact the decision whether to download or not. The factors that will need to be taken into consideration are: Network type, network "quality" (whatever that means), roaming status, battery status, user preference, and possibly more. E.g. on an EDGE network, when the user is roaming and almost out of battery, don't re-download the HD images if the user zoomed in.

WebKit doesn't change DPR and Firefox doesn't implement srcset, so blink would be the first one who has to deal with this.

 

Actually, this *should* also apply, more or less in the same way, to image-set.

IMO, the heuritics zoom-in part can be addressed at a later stage for both srcset and image-set. In the mean time zoom changes shouldn't re-download, just to be on the safe side.

[John Mellor]

Agree that this should affect initial resource selection, but that for now it's probably ok to ignore subsequent zoom changes (and we can tweak the heuristics later).

> deviceScaleFactor should already return the zoom impacted value.

No, the way this was implemented in crrev.com/23000005 and crrev.com/23192002, there is now a Frame::devicePixelRatio() that returns Page::deviceScaleFactor() * Frame::pageZoomFactor().

Several of the places that currently use Page::deviceScaleFactor(), should be changed to use Frame::devicePixelRatio(). I've left a comment on the patch :-)

[Yoav Weiss]

Thanks John! :) Will fix and re-upload.

[Rik Cabanier]

I agree. Just do it for the initial load and leave the heuristic for zooming in (with no reload) for later. 

[rosa maria palacios juncosa]