Possible implementation route towards <picture>

546 views
Skip to first unread message

Yoav Weiss

unread,
Jan 28, 2014, 6:17:35 PM1/28/14
to blink-dev
Hi,

Following Adam Barth's reply on the WHATWG list, I'd like to discuss implementation options for the picture element. I'm not yet certain that I'd be able to commit to implementing it, but I'd like to get a sense of what needs to be done. Once I'd know I can commit to it, I'll send a proper "Intent to implement". 

Since the biggest hurdle to full implementation is off-main-thread CSS parsing, the first phase is likely to exclude the syntax parts that rely on media queries. 
What would be the ideal fallback options for syntax that does include MQs? Will the fallback img@src be fetched and presented instead? (That seems like the simplest way to tackle this)

If I understand correctly, the first phase implementation requires the following:
* Avoid preloading of img@src & img@srcset resources when the img tag is inside an unclosed picture tag. Perform preloading that simulates picture's source selection algorithm in this case.
* Avoid immediate resource loading when img@src is parsed, and verify that the parent is not picture first. That will require insertion to the DOM before the resource can be loaded (if it wasn't already preloaded). If the parent is picture, proceed with picture's source selection algorithm.
* Implement the sizes attribute on both img and source (when it's under picture), but support only parsing of a single value (no breakpoints are allowed)
* Implement parsing of picture's extended srcset syntax, and base resource selection on it when sizes is present.

Regarding the second phase, implementing it requires off-main-thread MQ evaluation and (preferably) calc() evaluation. One way to achieve that is to wait long enough until CSSValue is overhauled, as part of the Oilpan effort. Adam has estimated the timeframe for this to happen in over a year, possibly two. 
Since this is a pretty long wait, I've been thinking of implementing a simpler CSS parser (possibly relying on Eric Seidel's tokenizer CL), that can focus on MQ evaluation and possibly calc().
Will such a "duplicate" parser be acceptable?
It can also be the initial step towards implementing a CSS parser that would eventually replace Bison. As a side effect of such a parser, it'd enable the preloadScanner to evaluate link@media internally, without joining the main thread, and enable hooking the preloadScanner directly to the networking thread without losing the savings that this evaluation provides.

Does any of this make sense? Am I missing any important roadblocks? Does this sound like a viable implementation plan?

Thanks!
Yoav

Eric Seidel

unread,
Jan 28, 2014, 6:31:05 PM1/28/14
to Yoav Weiss, blink-dev
On Tue, Jan 28, 2014 at 3:17 PM, Yoav Weiss <yo...@yoav.ws> wrote:
> Since the biggest hurdle to full implementation is off-main-thread CSS
> parsing, the first phase is likely to exclude the syntax parts that rely on
> media queries.
> What would be the ideal fallback options for syntax that does include MQs?
> Will the fallback img@src be fetched and presented instead? (That seems like
> the simplest way to tackle this)

Sounds reasonable to me. I don't feel super-well informed on this
discussion however.

> Regarding the second phase, implementing it requires off-main-thread MQ
> evaluation and (preferably) calc() evaluation. One way to achieve that is to
> wait long enough until CSSValue is overhauled, as part of the Oilpan effort.
> Adam has estimated the timeframe for this to happen in over a year, possibly
> two.
> Since this is a pretty long wait, I've been thinking of implementing a
> simpler CSS parser (possibly relying on Eric Seidel's tokenizer CL), that
> can focus on MQ evaluation and possibly calc().
> Will such a "duplicate" parser be acceptable?
> It can also be the initial step towards implementing a CSS parser that would
> eventually replace Bison. As a side effect of such a parser, it'd enable the
> preloadScanner to evaluate link@media internally, without joining the main
> thread, and enable hooking the preloadScanner directly to the networking
> thread without losing the savings that this evaluation provides.

We eventually need to sign up someone to actually do a real CSS Parser
re-write. But I don't believe that should block responsive images
work.

Adam Barth

unread,
Jan 28, 2014, 7:49:37 PM1/28/14
to Yoav Weiss, blink-dev
On Tue, Jan 28, 2014 at 3:17 PM, Yoav Weiss <yo...@yoav.ws> wrote:
Following Adam Barth's reply on the WHATWG list, I'd like to discuss implementation options for the picture element. I'm not yet certain that I'd be able to commit to implementing it, but I'd like to get a sense of what needs to be done. Once I'd know I can commit to it, I'll send a proper "Intent to implement". 

Since the biggest hurdle to full implementation is off-main-thread CSS parsing, the first phase is likely to exclude the syntax parts that rely on media queries. 
What would be the ideal fallback options for syntax that does include MQs? Will the fallback img@src be fetched and presented instead? (That seems like the simplest way to tackle this)

If I understand correctly, the first phase implementation requires the following:
* Avoid preloading of img@src & img@srcset resources when the img tag is inside an unclosed picture tag. Perform preloading that simulates picture's source selection algorithm in this case.
* Avoid immediate resource loading when img@src is parsed, and verify that the parent is not picture first. That will require insertion to the DOM before the resource can be loaded (if it wasn't already preloaded). If the parent is picture, proceed with picture's source selection algorithm.

Waiting to load <img> elements until they are inserted into the DOM isn't likely to be compatible with the web because many web sites that img elements created outside the DOM will start their network loads before being inserted in the DOM.  In fact, some web sites won't event insert the img elements into the DOM until their load event fires.  That's actually good for the user experience because that means the image will always appear in the next frame.
 
* Implement the sizes attribute on both img and source (when it's under picture), but support only parsing of a single value (no breakpoints are allowed)
* Implement parsing of picture's extended srcset syntax, and base resource selection on it when sizes is present.

Regarding the second phase, implementing it requires off-main-thread MQ evaluation and (preferably) calc() evaluation. One way to achieve that is to wait long enough until CSSValue is overhauled, as part of the Oilpan effort. Adam has estimated the timeframe for this to happen in over a year, possibly two. 
Since this is a pretty long wait, I've been thinking of implementing a simpler CSS parser (possibly relying on Eric Seidel's tokenizer CL), that can focus on MQ evaluation and possibly calc().
Will such a "duplicate" parser be acceptable?

We have some simplified CSS parsers for some performance critical code paths.  I'm not sure how much of the CSS language this simplified parser would need to parse.  If you need to parse constructs like calc(), you'll pretty much need a full lexer...
 
It can also be the initial step towards implementing a CSS parser that would eventually replace Bison. As a side effect of such a parser, it'd enable the preloadScanner to evaluate link@media internally, without joining the main thread, and enable hooking the preloadScanner directly to the networking thread without losing the savings that this evaluation provides.

That seems valuable.  We'd also like to be able to do some simple CSS selector matching during preloading, for example to preload background-images for elements that match a class selector.  It's like we could make that work if we had a threadsafe lexer.

Does any of this make sense? Am I missing any important roadblocks? Does this sound like a viable implementation plan?

That basically sounds like a viable plan.  There are some details to work out regarding the compatibility impact and whether we'll want to have two CSS lexers.

I wonder if we can also make incremental progress by shipping the img@srcset implementation we already have in the tree.  It seems like that's a subset of the full proposal that might make sense to ship in parallel with working on the rest of the feature.

Adam

John Mellor

unread,
Jan 29, 2014, 6:40:25 AM1/29/14
to Adam Barth, Yoav Weiss, blink-dev
On 29 January 2014 00:49, Adam Barth <aba...@chromium.org> wrote:
On Tue, Jan 28, 2014 at 3:17 PM, Yoav Weiss <yo...@yoav.ws> wrote:
* Avoid immediate resource loading when img@src is parsed, and verify that the parent is not picture first. That will require insertion to the DOM before the resource can be loaded (if it wasn't already preloaded). If the parent is picture, proceed with picture's source selection algorithm.

Waiting to load <img> elements until they are inserted into the DOM isn't likely to be compatible with the web because many web sites that img elements created outside the DOM will start their network loads before being inserted in the DOM.  In fact, some web sites won't event insert the img elements into the DOM until their load event fires.  That's actually good for the user experience because that means the image will always appear in the next frame.

Yes, a common pattern in JS is:  img = new Image(); img.onload = foo; img.src = url;
Perhaps a compromise would work: when src is set on a image, post a task to the message loop that will start loading the src attribute, unless in the meantime it has been added to the DOM as a child of a picture element. We'd need to check that this doesn't regress performance too much.

Yoav Weiss

unread,
Jan 29, 2014, 7:01:39 AM1/29/14
to John Mellor, Adam Barth, blink-dev
On Wed, Jan 29, 2014 at 12:40 PM, John Mellor <joh...@chromium.org> wrote:
On 29 January 2014 00:49, Adam Barth <aba...@chromium.org> wrote:
On Tue, Jan 28, 2014 at 3:17 PM, Yoav Weiss <yo...@yoav.ws> wrote:
* Avoid immediate resource loading when img@src is parsed, and verify that the parent is not picture first. That will require insertion to the DOM before the resource can be loaded (if it wasn't already preloaded). If the parent is picture, proceed with picture's source selection algorithm.

Waiting to load <img> elements until they are inserted into the DOM isn't likely to be compatible with the web because many web sites that img elements created outside the DOM will start their network loads before being inserted in the DOM.  In fact, some web sites won't event insert the img elements into the DOM until their load event fires.  That's actually good for the user experience because that means the image will always appear in the next frame.

Yes, a common pattern in JS is:  img = new Image(); img.onload = foo; img.src = url;
 
Yes, I'm aware of this pattern and agree this is the desired and expected behavior.  
 
Perhaps a compromise would work: when src is set on a image, post a task to the message loop that will start loading the src attribute, unless in the meantime it has been added to the DOM as a child of a picture element. We'd need to check that this doesn't regress performance too much.


I took a slightly different approach in a local branch, basically "marking" the element when an attribute is set by JS, thus enable img@src resource loading only when the attribute was set that way.
I'm not certain my approach is valid (e.g. each attribute set by JS now has to set this boolean, which is not ideal), but it seems to work. I've now uploaded it, and am running perf tests, to see its implications. 


Yoav Weiss

unread,
Jan 29, 2014, 8:04:14 AM1/29/14
to John Mellor, Adam Barth, blink-dev
On Wed, Jan 29, 2014 at 1:01 PM, Yoav Weiss <yo...@yoav.ws> wrote:

I took a slightly different approach in a local branch, basically "marking" the element when an attribute is set by JS, thus enable img@src resource loading only when the attribute was set that way.
I'm not certain my approach is valid (e.g. each attribute set by JS now has to set this boolean, which is not ideal), but it seems to work. I've now uploaded it, and am running perf tests, to see its implications. 

 
My approach seems to break innerHTML related tests on the bots, so it's definitely invalid. Please ignore the CL.

Simon Pieters

unread,
Jan 31, 2014, 3:22:18 AM1/31/14
to John Mellor, Yoav Weiss, Adam Barth, blink-dev
> it <https://codereview.chromium.org/135273010>, and am running perf
> tests,
> to see its implications.

How to implement this needs to be specified, which is
https://github.com/ResponsiveImagesCG/picture-element/issues/91

--
Simon Pieters
Opera Software

Yoav Weiss

unread,
Jan 31, 2014, 3:48:54 AM1/31/14
to Simon Pieters, John Mellor, Adam Barth, blink-dev
I agree that the "what to implement" parts should be specified. IMO we should spec:
* JS based 'src' setting should trigger an immediate download, before the element is added to the DOM
* Parser based 'src' setting shouldn't trigger an immediate download. The full selection algorithm should be triggered once HTMLImageElement is added to the DOM. 
* innerHTML handling is slightly more complicated - we should probably specify that selection algorithm is triggered once the innerHTML section is added to the parent element, if it wasn't already triggered during parsing of the innerHTML value.

I'll add this to the GitHub issue, and we should probably continue the "what" discussion there.

OTOH, "how to implement" the above without hurting performance of existing content is a different discussion IMO.

--
Simon Pieters
Opera Software


To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Adam Barth

unread,
Jan 31, 2014, 4:06:00 AM1/31/14
to Yoav Weiss, Simon Pieters, John Mellor, blink-dev
On Fri, Jan 31, 2014 at 12:48 AM, Yoav Weiss <yo...@yoav.ws> wrote:
I agree that the "what to implement" parts should be specified. IMO we should spec:
* JS based 'src' setting should trigger an immediate download, before the element is added to the DOM
* Parser based 'src' setting shouldn't trigger an immediate download. The full selection algorithm should be triggered once HTMLImageElement is added to the DOM. 

If you special-case the parser like that, won't that make it difficult to implement an HTML parser in JavaScript?  (See https://docs.google.com/a/chromium.org/document/d/1ZkV1PpPsJJgdSZOA10Jh0VrThR6D_Q0XWv_2B9-0gGE/edit?pli=1 for context.)

Adam

Yoav Weiss

unread,
Jan 31, 2014, 4:47:55 AM1/31/14
to Adam Barth, Simon Pieters, John Mellor, blink-dev
Hmm. It probably will. We could possibly resolve this by adding a JS API to signify "I am a parser so don't immediately load resources" upon the HTMLImageElement creation or shortly after, but I won't argue that this solution is not ugly.
Alternatively, we could avoid special-casing and have the parser maintain some state that keeps track of <picture> & </picture> tokens and notifies HTMLImageElement of that state upon creation. I'm not sure that is cleaner.
 
Adam


Simon Pieters

unread,
Jan 31, 2014, 5:14:24 AM1/31/14
to Yoav Weiss, John Mellor, Adam Barth, blink-dev
On Fri, 31 Jan 2014 09:48:54 +0100, Yoav Weiss <yo...@yoav.ws> wrote:

>> How to implement this needs to be specified, which is
>> https://github.com/
>> ResponsiveImagesCG/picture-element/issues/91
>>
> I agree that the "what to implement" parts should be specified. IMO we
> should spec:
> * JS based 'src' setting should trigger an immediate download, before the
> element is added to the DOM

"immediate" is wrong per spec even ignoring <picture>.
http://www.whatwg.org/specs/web-apps/current-work/multipage/embedded-content-1.html#update-the-image-data
step 8 requires you to await a stable state, which basically means to let
the current script finish or, in the HTML parser case, wait until the
parser yields to the event loop (or encounters a </script> tag which
provides a stable state).

The spec requires this so that authors can set the src and crossorigin
attributes in any order in script and it should have the same effect.

I think the same reasoning should apply to script-created img/picture
structure: you don't want to start downloading a script-created img if the
next line in the script is going to append it to a picture element, so we
should await a stable state.

Yoav Weiss

unread,
Jan 31, 2014, 4:14:45 PM1/31/14
to Adam Barth, blink-dev
On Wed, Jan 29, 2014 at 1:49 AM, Adam Barth <aba...@chromium.org> wrote:

I wonder if we can also make incremental progress by shipping the img@srcset implementation we already have in the tree.  It seems like that's a subset of the full proposal that might make sense to ship in parallel with working on the rest of the feature.

 
That makes sense. I'll send out an "intent to ship". 

Christian Biesinger

unread,
Feb 13, 2014, 5:21:46 PM2/13/14
to Yoav Weiss, Adam Barth, blink-dev
Hi Yoav,

I just wanted to know what the current status of <picture> is, and if
there's any way I can help with the implementation/shipping of this
feature. Let me know!

-christian

Yoav Weiss

unread,
Feb 14, 2014, 10:15:28 AM2/14/14
to Christian Biesinger, Adam Barth, blink-dev
On Thu, Feb 13, 2014 at 11:21 PM, Christian Biesinger <cbies...@chromium.org> wrote:

Hi Yoav,

I just wanted to know what the current status of <picture> is, and if
there's any way I can help with the implementation/shipping of this
feature. Let me know!

 
Hi Christian,

Help will be highly appreciated :)

Currently there are several issues that need implementing:
* Thread safe MQ parser - I'm currently working on that on a local branch, based on eseidel's new tokenizer. I've got the tokenizer working (at least) for simple MQ syntax, and now I need to get the parser that uses it in shape. I hope I'll have something for review after the weekend.
* <img> resource loading to wait for stable state - Philip J is working on creating a stable state for Media elements that we can use once it lands. I think that most of the work here would be to verify that resource loading doesn't break with this change, and that it causes no performance regressions. I already added some tests to prevent regressions here, but I'm not sure I covered everything. I think that the onload event needs some further tests to make sure its behavior doesn't change.
* Add the picture element and adapt source
* Extend srcset to include resource dimensions
* Implement sizes parsing
* Implement the resource selection algorithm
* Modify the preload scanner to play nice with all that.

If you could handle the "img resource loading" part at first, I would highly appreciate it, since it's important to get it right for the sake of the rest of the implementation (and to avoid breaking the Web), and I'm currently focused on MQ evaluation.
Once both resource loading and MQ evaluation are out of the way (and an "intent to implement" is sent), we could probably split the rest of the tasks between us.


augusto...@gmail.com

unread,
Sep 3, 2014, 7:57:31 AM9/3/14
to blin...@chromium.org
Hi, I think that it could be better if the tag is more "generic" to use it with any type of multimedia content like videos, sounds, etc (low quality versions), or tables, texts, etc (shorter or smaller versions). For example: <content>

Sorry for my english.

Thank you. Bye.
Reply all
Reply to author
Forward
0 new messages