[blink-dev] Intent to Implement: picture element

[Yoav Weiss]

Emails

Yoav Weiss - yo...@yoav.ws

Spec

http://www.w3.org/TR/html-picture-element/

Summary

Enable a responsive images solution by giving developers control over the image resource that is downloaded & displayed using multiple resources, each with its own media attributes. 

Motivation

Responsive Web Design is the new hotness, and it has many advantages when developing Web sites to be viewed across a large variety of screen sizes and resolutions. One of the major criticisms of RWD is its performance, and the fact that it doesn't take advantage of smaller screen sizes to reduce the site's bytesize. Since images are 60% of the average Web site's bytesize, resolving the responsive images problem and serve images in different dimensions and proportions to different screen viewports and resolutions is a big step towards resolving that performance issue. It will also enable Web developers to adapt the images they use to their responsive layout.

Compatibility Risk

The specification is a working draft published 2 months ago. The feature has a popular polyfill used by numerous large sites. Other browser vendors haven't yet expressed intentions to implement the specification.

OWP launch tracking bug?

http://crbug.com/233751

Row on feature dashboard?

Yes

Requesting simultaneous permission to ship?

No

[Elliott Sprehn]

Sounds like a cool feature.

On Sun, Apr 21, 2013 at 1:35 PM, Yoav Weiss <yo...@yoav.ws> wrote:

...

Compatibility Risk

The specification is a working draft published 2 months ago. The feature has a popular polyfill used by numerous large sites. Other browser vendors haven't yet expressed intentions to implement the specification.

What large sites? :)

-E

[Yoav Weiss]

Thanks! :)

A few large sites I'm aware of are microsoft.com, dribbble.com, flavorwire.com & globalnews.ca.

[Darin Fisher]

Isn't <picture> redundant with <img> and image-set?  What am I missing?

[Dirk Schulze]

On Apr 21, 2013, at 8:23 PM, Darin Fisher <da...@chromium.org> wrote:

> Isn't <picture> redundant with <img> and image-set? What am I missing?

image-set and <picture> can be seen as two concurring proposals for a similar/the same goal.

Greetings,
Dirk

[Ojan Vafai]

On Sun, Apr 21, 2013 at 8:52 PM, Dirk Schulze <dsch...@chromium.org> wrote:

On Apr 21, 2013, at 8:23 PM, Darin Fisher <da...@chromium.org> wrote:

> Isn't <picture> redundant with <img> and image-set?  What am I missing?

image-set and <picture> can be seen as two concurring proposals for a similar/the same goal.

Given that we're already shipping image-set, we'd need significant use-cases to justify also shipping <picture>. The spec draws a comparison to srcset as well, stating that <picture> is needed to meet some responsive image use-cases that srcset doesn't meet, but doesn't give specifics. Are there use-cases that aren't met by some combination of srcset, CSS media queries and image-set? Is the only non-syntatic difference that the UA is not allowed to pick which resource it wants to use with <picture>?

[Yoav Weiss]

On Mon, Apr 22, 2013 at 6:15 AM, Ojan Vafai <ojan@chromium.org> wrote:

On Sun, Apr 21, 2013 at 8:52 PM, Dirk Schulze <dsch...@chromium.org> wrote:

On Apr 21, 2013, at 8:23 PM, Darin Fisher <da...@chromium.org> wrote:

> Isn't <picture> redundant with <img> and image-set?  What am I missing?

image-set and <picture> can be seen as two concurring proposals for a similar/the same goal.

Image-set resolves the problem of DPR based resolution switching for CSS images. It does not resolve that problem for content images.
Both <picture> and srcset and targeted to resolve the responsive images problem for content images.
 

Given that we're already shipping image-set, we'd need significant use-cases to justify also shipping <picture>. The spec draws a comparison to srcset as well, stating that <picture> is needed to meet some responsive image use-cases that srcset doesn't meet, but doesn't give specifics. Are there use-cases that aren't met by some combination of srcset, CSS media queries and image-set? Is the only non-syntatic difference that the UA is not allowed to pick which resource it wants to use with <picture>?

That is in fact one of the main differences between srcset and picture, along with the fact that <picture> can define any media attribute as means to pick a resource, and the fact that <picture> can define a `type` attribute on each resource, which provides means for client-side content negotiation.
A full specification of the use-cases for responsive images can be found at: http://www.w3.org/TR/respimg-usecases/
The ones that <picture> satisfies well and srcset falls short are:
* Art-direction
* Matching media features
* Image formats

OTOH, for the resolution switching use-case, srcset is superior since it lets the UA optimize and avoid the download when it is not necessary (higher res image was already downloaded), cannot be achieved (poor network connectivity), or that the user specified a preference to sacrifice visual quality in favor of bandwidth savings.

So, I don't think any of these specs is concurring with image-set, nor are they concurring with each other. I believe these specs are complementary.

[Adam Barth]

I don't think we should implement the picture element at this time.  Unlike some of the other proposals we've been discussing on this mailing list, <picture> needs to integrate tightly with Core and therefore implementing <picture> will constrain our ability to improve Core in the future.

For example, we're working on having the HTMLPreloadScanner issue preload requests from the parser thread (i.e., without needing to joining the main thread).  In order for the HTMLPreloadScanner to issue preload requests for <picture> elements, the HTMLPreloadScanner would need to be able to evaluate arbitrary media requests.  That's difficult to do without joining the main thread because the media query engine works only on the main thread.

It's possible that we'll want to implement <picture> in the future, but, at the moment, I don't think its value outweighs the constraints it imposes on future development of Core.

Adam

[rob.wie...@gmail.com]

Some other large projects implementing picturefill: National Geographic Magazine, the Drupal 8 cms. Picture has near complete buy in from responsive web developers.

[Yoav Weiss]

Regarding the specific HTMLPreloadScanner issue you're raising, I believe we can use the same technique used in https://codereview.chromium.org/13945017/ to enable `media` attribute based preloading without giving the PreloadScanner the ability to evaluate MQs. I have working code that does that, and I'll submit it for review shortly.

Are there are other aspects of <picture> support that may affect Core's future improvement?

Any true responsive images solution will require some form of MQ evaluation (at least a subset of the range of possible MQs) in order to preload only the required resources. I can't evaluate the future costs, but I think the value of a responsive images native solution is significant, since 60% of traffic is images and a large portion of that traffic can be saved on lower resolution devices.

[Adam Barth]

On Mon, Apr 22, 2013 at 9:53 AM, Yoav Weiss <yo...@yoav.ws> wrote:

Regarding the specific HTMLPreloadScanner issue you're raising, I believe we can use the same technique used in https://codereview.chromium.org/13945017/ to enable `media` attribute based preloading without giving the PreloadScanner the ability to evaluate MQs. I have working code that does that, and I'll submit it for review shortly.

That CL means that we can't issue stylesheet preloads without joining the main thread.  That's unfortunately, but unavoidable given the API commitments we've already made.  If we implemented <picture>, we would be in a similar predicament with images.

It's not that these problems aren't solvable, it's just that they slow down other improvements.  Adding features to the engine isn't free, especially when they're tightly coupled with the rest of Core.

Are there are other aspects of <picture> support that may affect Core's future improvement?

As another example, we're in the process of moving image decoding off the main thread unto the rasterization thread.  Does <picture> make that work more difficult?  It's not entirely clear to me, but it seems possible given that <picture> breaks some assumptions that would otherwise be true of <img>.

Any true responsive images solution will require some form of MQ evaluation (at least a subset of the range of possible MQs) in order to preload only the required resources. I can't evaluate the future costs, but I think the value of a responsive images native solution is significant, since 60% of traffic is images and a large portion of that traffic can be saved on lower resolution devices.

There are a number of competing proposals in this area, and it's not clear to me yet which ones make the best trade-offs.  For example, <https://github.com/igrigorik/http-client-hints> is a proposal for using HTTP content negotiation rather than media queries to address these use cases.

I know there are many pros and cons to these different proposals and there have been epicly long threads in the W3C on these topics.  I'd like to try to avoid starting a long debate weighing the pros and cons on this mailing list.  It's better to have that conversation in the W3C with other implementers participating.

[mathew...@gmail.com]

On Monday, April 22, 2013 1:34:02 PM UTC-4, Adam Barth wrote:

On Mon, Apr 22, 2013 at 9:53 AM, Yoav Weiss <yo...@yoav.ws> wrote:

Any true responsive images solution will require some form of MQ evaluation (at least a subset of the range of possible MQs) in order to preload only the required resources. I can't evaluate the future costs, but I think the value of a responsive images native solution is significant, since 60% of traffic is images and a large portion of that traffic can be saved on lower resolution devices.

There are a number of competing proposals in this area, and it's not clear to me yet which ones make the best trade-offs.  For example, <https://github.com/igrigorik/http-client-hints> is a proposal for using HTTP content negotiation rather than media queries to address these use cases.

I know there are many pros and cons to these different proposals and there have been epicly long threads in the W3C on these topics.  I'd like to try to avoid starting a long debate weighing the pros and cons on this mailing list.  It's better to have that conversation in the W3C with other implementers participating.

I just want to stress again that the existing discussions in the HTML WG—and both the `picture` and `srcset` FPWDs—do not frame the proposals as competing, but complimentary. This has been the case for quite some time, despite the original “choose one or the other” nature of the two proposals as they were first introduced. As you said, those early conversations took place at tremendous length, and the conclusion was that there are use cases that are best (and only) addressed by `srcset` and use cases that are best (and only) addressed by `picture`. The full list of Use Cases and Requirements—also published by the HTML WG, without objection—is addressed by both solutions either individually or in concert.

Also, I don’t believe the client hints proposal has been put forth as addressing “responsive images” use cases, though I may be mistaken. The technical limitations around client hints containing viewport information would likely make unsuitable for this purpose. 

-M

[Michael[tm] Smith]

On Tue, Apr 23, 2013 at 2:51 AM, <mathew...@gmail.com> wrote:

> I just want to stress again that the existing discussions in the HTML WG—and
> both the `picture` and `srcset` FPWDs—do not frame the proposals as
> competing, but complimentary. This has been the case for quite some time,
> despite the original “choose one or the other” nature of the two proposals
> as they were first introduced. As you said, those early conversations took
> place at tremendous length, and the conclusion was that there are use cases
> that are best (and only) addressed by `srcset` and use cases that are best
> (and only) addressed by `picture`.

Speaking as somebody who followed those discussions, I agree that all
the above is an accurate assessment. And that's not just having
followed the HTML WG discussions, but the Responsive Images Community
Group discussions before that, and the discussions on the WHATWG
mailing list before that.

> The full list of Use Cases and
> Requirements—also published by the HTML WG

At http://www.w3.org/TR/respimg-usecases/

[Dimitri Glazkov]

FWIW, I think the problem you're trying to solve is interesting and worthy of tackling. I would highly recommending expanding your group of problem-solvers to a few browser folks, who can help out with ironing out the implementation kinks -- and as a bonus, get a better chance of browser vendor commitment.

:DG<

[Eric Seidel]

I also think this technology is interesting.  I'm also sad that we appear to be using a not-yet-standardized "image-set" CSS property to do so:

http://dev.w3.org/csswg/css-images/#image-set-notation

I'm glad this has a good polyfill.  I think that's a great way for features to evolve when possible.

However, given the objections from two API reviewers on this list, the implement recommendation is currently:

NOT LGTM

Blink has an overlapping/competing-standard implementation and is not ready to commit to this feature. I think we should re-evaluate how standards and other browsers have evolved here again in the future.

[Ojan Vafai]

As far as API details, I think you'd have better traction on this (both with Blink and other browser vendors) if you could design the API to be built on top of <img> and srcset/image-set instead of trying to add a new HTML element for loading images.

[Yoav Weiss]

Thank you all for your comments.

I understand that the NOT LGTM decision is final, at least for the time being. I would love to understand with some more detail the reasons that have led to this decision.

I have submitted my current <picture> implementation to https://codereview.chromium.org/14449003 . A review of this patch (regardless of the fact that the feature will not be included) would be highly appreciated. 

Finally, we (the RICG) would highly welcome any browser people that are interested in getting involved in resolving the responsive images problem, be it using the picture element, srcset or any other means that resolve the set of use cases we have gathered.

[Eric Seidel]

My understanding of the discussion is that <picture> differs from
image-set in that it uses arbitrary media queries, which is
significantly more complicated than image-set. This increases the
implementation complexity.

You implementation was nice, btw, but highlights the added complexity
in the preload scanner. I believe mis-nested <picture> tags could
confuse your current implementation into ignoring preloads for the
rest of the document?


Since "intent to implement" emails are more advisory than
permission-seeking, I would take the "NOT LGTM" as just a strong
signal that at current you're going to have a difficult time finding
OWNERS to "lgtm" your patch (rather than a final decision that Blink
will never implement this).

I think we should focus on making our current implementations of
srcset and image-set awesome enough to answer developer's (and any
current polyfill's) needs and re-consider adding <picture> if the
landscape changes.

[Yoav Weiss]

On Wed, Apr 24, 2013 at 2:09 AM, Eric Seidel <ese...@chromium.org> wrote:

My understanding of the discussion is that <picture> differs from
image-set in that it uses arbitrary media queries, which is
significantly more complicated than image-set.  This increases the
implementation complexity.

You implementation was nice, btw, but highlights the added complexity
in the preload scanner.  I believe mis-nested <picture> tags could
confuse your current implementation into ignoring preloads for the
rest of the document?

It is highly possible that my current (initial) implementation is not perfect :) What exactly is the scenario you see as problematic? 

Since "intent to implement" emails are more advisory than
permission-seeking, I would take the "NOT LGTM" as just a strong
signal that at current you're going to have a difficult time finding
OWNERS to "lgtm" your patch (rather than a final decision that Blink
will never implement this).

I think we should focus on making our current implementations of
srcset and image-set awesome enough to answer developer's (and any
current polyfill's) needs and re-consider adding <picture> if the
landscape changes.

srcset is not currently implemented AFAIK. If it would be, it'd be awesome, since it'd resolve some of the major use-cases for responsive images, including resolution-switching which is extremely important.
I think srcset's implementation will struggle at least with some of the same problems, since it'd have to calculate the viewport and DPR in order to figure out the required resource to preload. It'll probably be slightly easier, since we can copy the entire attribute's value to the main thread and take the decision there.


 

[Rick Byers]

We considered implementing srcset for the Chromebook pixel launch, but realized that (for the common scenarios at least) it was trivially implementable on top of -webkit-image-set, eg.:

   <img style="display: inline-block; content: -webkit-image-set(url('foo.png') 1x, url('foo_2x.png') 2x);">

That said, I think it's high time that we settled on the standard definition of imgset and srcset and implemented them in Blink rather than rely on webkit specific APIs.  I'd love to see an 'intent to implement' for that ;-)

Rick

[Eric Seidel]

Since bitmap images aren't going away, and devices are clearly getting
more pixels (at least in the short term) I can see an obvious need for
authors to want to vend the correct bitmap to the correct device.
Which variant of these features is the right one is not clear at this
time. But the one which is the least complicated to implement
(image-set) has won for the time being.

There are plenty of improvements we could make to the existing code if
you're interested in hacking in this area.

While reviewing the CSSPreloadScanner code before replying to your
initial email, I found that we don't seem to handle image-set preloads
particularly well:
https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/core/html/parser/CSSPreloadScanner.cpp&q=CSSPreloadScanner&sq=package:chromium&type=cs&l=29
I believe that we either preload all of variants of an image-set, or
none of them. We'd have to test to confirm, but neither is ideal.

Any work in this area should be aware of the fact that *very soon* the
HTML Parser Thread is going to start sending preloads directly to the
loader thread, bypassing the main thread (which currently gets the
preloads delivered to it, only to bounce them off to the loader
thread). image-set won't get in the way of this because we can just
tell either thread which image variant (1x or 2x) we should be
loading. For <picture> (as defined) this is not possible w/o sending
the preloads through the main thread. This is not an absolute
deal-breaker for this feature, but is complexity we'd like to avoid
adding for this (not yet widely used feature) to code that is under
active revision at this time.

[Bruce Lawson]

On Monday, 22 April 2013 04:23:34 UTC+1, Darin Fisher wrote:

Isn't <picture> redundant with <img> and image-set?  What am I missing?

One use for <picture> is to allow devs to send webP images to capable browsers, with fallback to PNG/ whatevs for non-webP browsers (I expound more at http://www.brucelawson.co.uk/2013/save-bandwidth-webp-with-fallback/) .

(I don't think anyone's married to <picture> as an element name, but a simple declarative way to address this and the art-direction responsive images use case is a problem that needs solving), in my opinion (and in the opinion of many devs I talk to).

(Sent from private email account as I'm not representing Opera here)

Bruce Lawson

[Adam Barth]

On Fri, Apr 26, 2013 at 10:01 AM, Bruce Lawson <bruce.c...@gmail.com> wrote:

On Monday, 22 April 2013 04:23:34 UTC+1, Darin Fisher wrote:

Isn't <picture> redundant with <img> and image-set?  What am I missing?

One use for <picture> is to allow devs to send webP images to capable browsers, with fallback to PNG/ whatevs for non-webP browsers (I expound more at http://www.brucelawson.co.uk/2013/save-bandwidth-webp-with-fallback/) .

I'm hopeful that we'll be able to use the Accept header and HTTP content negotiation for that purpose.  That way servers and network intermediaries can use send WebP to supporting user agents without authors needing to modify their content.

Adam

[Yoav Weiss]

Thanks for the detailed explanation. 

I think that any PreloadScanner modifications should take into account that external stylesheets are currently preloaded according to their media attribute. <picture>, srcset or any other responsive images solution will need to take viewport dimensions and DPR (at the very least) into account. Other resources, mainly JS, can also benefit from loading according to a media query in the future.

RWD, which relies on media queries to determine the page's layout and content, needs corresponding loading mechanisms. In their absence, Web developers that want to load only the required resources are forced to "cheat" the PreloadScanner by hiding their resources in JS, or as data attributes. This often results in delays to the resource loading that can be avoided, if the native loading mechanisms would take media attributes into account. It also results in mis-information aregarding the benefits and purpose of preloaders.

So, regardless of <picture> specifically, please leave an option in the new architecture for media based loading.