Roadmap Q2 goal : Pass Acid3

[David Bruant]

Hi,

If I may weigh in, I wonder if passing Acid3 is a valuable goal in the
short term.
IE8 scores 20/100 at Acid3. All major websites and most websites have to
support IE8, because it's the current "boat-anchor browser" [1]. So most
websites don't need the features that make a 100 score.
Specifically, SVG might be a big piece and since IE8 doesn't support it,
usage is still very rare in websites.

Aiming at IE8 parity (... or rather the intersection of IE8 and current
standards, obviously) feels like a more practical goal if you want to
support the web as it is today. An incomplete list of relevant features
to get to IE8 parity [2].

Said differently, aiming at IE8 parity can only yield a much much better
supported-website-count/effort ratio than passing Acid3.

David

[1] http://infrequently.org/2010/10/ie-8-is-the-new-ie-6/
[2] http://caniuse.com/#compare=ie+8

[Robert O'Callahan]

I don't think we should be setting these kinds of goals for Servo in the
short term. Passing Acid-style tests or meeting site-compatibility criteria
require doing a lot of work that is not particularly risky and does not
have much architectural impact on the engine ... but there's a lot of work
to do that is risky and does have significant architectural impact, and
that work should be higher priority, IMHO.

Rob
--
Jtehsauts tshaei dS,o n" Wohfy Mdaon yhoaus eanuttehrotraiitny eovni
le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o Whhei csha iids teoa
stiheer :p atroa lsyazye,d 'mYaonu,r "sGients uapr,e tfaokreg iyvoeunr,
'm aotr atnod sgaoy ,h o'mGee.t" uTph eann dt hwea lmka'n? gBoutt uIp
waanndt wyeonut thoo mken.o w

[James Graham]

On 25/02/14 11:42, David Bruant wrote:
> Hi,
>
> If I may weigh in, I wonder if passing Acid3 is a valuable goal in the
> short term.
> IE8 scores 20/100 at Acid3. All major websites and most websites have to
> support IE8, because it's the current "boat-anchor browser" [1]. So most
> websites don't need the features that make a 100 score.
> Specifically, SVG might be a big piece and since IE8 doesn't support it,
> usage is still very rare in websites.
>
> Aiming at IE8 parity (... or rather the intersection of IE8 and current
> standards, obviously) feels like a more practical goal if you want to
> support the web as it is today. An incomplete list of relevant features
> to get to IE8 parity [2].
>
> Said differently, aiming at IE8 parity can only yield a much much better
> supported-website-count/effort ratio than passing Acid3.

I think I agree that passing Acid 3 in the next four months is not going
to happen. But that's more because it requires a whole set of
technologies that we don't have an implementation of in Servo (SVG, DOM
Range+traversal, etc.) and which don't obviously seem like high
priorities compared to getting the fundamental architecture right.

I also think that the acid tests in general and acid 3 in particular are
rather flawed tests; they make great PR pieces and that's a good reason
to target them, but the actual things that they test are a grab-bag of
random features and implementation bugs from the time they were written.

Having said that, I don't think that using "IE8 parity" as a metric is
very useful. IE8 takes a lot of non-standard codepaths on real sites, so
merely targeting the standard features it implements is not going to
give you an end result that is actually compatible with all the sites
that work with IE8 (for example, I believe that IE8 still has the old
attachEvent junk rather than the standard addEventListener API, so by a
literal reading of the above you wouldn't implement events at all).

I think a better way to proceed is to try to implement the parts that
change the underlying architecture as early as possible (so, on the DOM
side, this would be HTML parsing, page loading, script scheduling,
history navigation, etc.) and as a minimum target as-good-as-gecko (or
webkit or blink, or whatever) scores on the relevant web-platform-tests
for those features (in practice we want to aim for 100% of course, but
as long as we at least pass the tests that everyone else passes the
compat. story should be OK). Once this is done we survey web content to
see which higher-level features are most used and implement those in
priority order using the same approach.

My method isn't likely to produce the best short-term gains, but I think
it's the best route to compatibility with real sites in the long term.

[Boris Zbarsky]

On 2/25/14 6:42 AM, David Bruant wrote:
> If I may weigh in, I wonder if passing Acid3 is a valuable goal in the
> short term.

Absolutely not. It's not a valuable goal in any term, really, until
we're trying to go for random "standards pr" points.

-Boris

[David Bruant]

I forgot. I was referring to https://github.com/mozilla/servo/wiki/Roadmap

Le 25/02/2014 12:57, James Graham a écrit :
> Having said that, I don't think that using "IE8 parity" as a metric is
> very useful. IE8 takes a lot of non-standard codepaths on real sites,
> so merely targeting the standard features it implements is not going
> to give you an end result that is actually compatible with all the
> sites that work with IE8 (for example, I believe that IE8 still has
> the old attachEvent junk rather than the standard addEventListener
> API, so by a literal reading of the above you wouldn't implement
> events at all).

blah :-p Yes of course, don't do a literal reading. What I tried to
convey takes lots of words if aiming at accuracy and I feel "IE8 parity"
roughly conveys it. Among other things, I'm obviously not recommanding
to remove features that are already in Servo, but not in IE8 ;-)
My main point was about the supported-website-count/effort ratio. And
that aiming at some IE8 features may be more relevant than Acid3
features, especially looking at the second overarching goal ("render
most web pages faithfully") and Q4 goal.

> I think a better way to proceed is to try to implement the parts that
> change the underlying architecture as early as possible (so, on the
> DOM side, this would be HTML parsing, page loading, script scheduling,
> history navigation, etc.)

I agree with you and roc on this part and didn't mention it because I
agree. Only the Acid3 part felt a bit off (at least for now. I
acknowledge the PR value, but it doesn't seem like a concern for 2014)

David

[Jack Moffitt]

>> If I may weigh in, I wonder if passing Acid3 is a valuable goal in the
>> short term.
>

> Absolutely not. It's not a valuable goal in any term, really, until we're
> trying to go for random "standards pr" points.

Acid2 and Acid3 are the current goals of our partners, and the Servo
team is helping them achieve those goals. This is helping design and
reviewing code. It's not insignificant work, but we think having
partners is important. You can ping me off list for more info and
context if you would like.

The MoCo employees working on Servo are currently focusing on the rest
of those items in the list.

jack.

[David Bruant]

Le 25/02/2014 17:14, Jack Moffitt a écrit :
>>> If I may weigh in, I wonder if passing Acid3 is a valuable goal in the
>>> short term.

>> Absolutely not. It's not a valuable goal in any term, really, until we're
>> trying to go for random "standards pr" points.
> Acid2 and Acid3 are the current goals of our partners, and the Servo
> team is helping them achieve those goals. This is helping design and
> reviewing code. It's not insignificant work, but we think having
> partners is important.

Agreed. I wasn't aware of this context. Withdrawing my doubts about Acid3.

Thanks,

David

[Patrick Walton]

On 2/25/14 3:55 AM, Robert O'Callahan wrote:
> I don't think we should be setting these kinds of goals for Servo in the
> short term. Passing Acid-style tests or meeting site-compatibility criteria
> require doing a lot of work that is not particularly risky and does not
> have much architectural impact on the engine ... but there's a lot of work
> to do that is risky and does have significant architectural impact, and
> that work should be higher priority, IMHO.

If I may defend the Acid tests just a little bit...

From my point of view, especially with my work lately, one of the most
important questions that Servo needs to answer is "how much parallelism
can we gain on real-world Web sites?" To that end, Web site
compatibility (which Acid2 is a rough proxy for a large subset of, if
used sensibly [1]) is a useful thing to work on. Otherwise, we don't
know whether the parallel gains that we are getting will scale up to
real-world usage, or whether we're only getting them because we don't
implement the difficult features.

To give a concrete example, the work on block formatting contexts [2]
and absolute positioning [3] has been hugely illustrative. Not only was
this work a sea change to our layout algorithm, it helped us establish
exactly what the parallel hazards with floats were. At the beginning
before we implemented floats, we got massive parallel gains on all
sites, but these were unrealistic. In Servo master, we're using a
conservative algorithm to handle floats that completely erases any
parallel gains on most sites. Only with this block formatting context
work--which is also necessary for Web compat--do we start to see gains
on most sites. Handling absolute positioning properly will let us
perform further measurements on how much parallelism is impacted on
sites: in particular, how often in the real world is an absolute frame's
hypothetical box impacted by floats? We don't know the answer to that
right now, but this seems to me to be a very important question.

So I agree that we must not make architectural decisions that make it
difficult to implement Web features. But I do think that Web compat is
useful, because it helps us to identify the parallel hazards and know
where we stand in terms of the Web of today, which helps us answer the
big questions that Servo set out to answer.

Patrick

[1]: "Sensibly" means implementing the features that the test tests in a
way that handles even edge cases that are not tested, and not accepting
patches that would make it hard implement more complex features that are
not tested, such as incremental reflow, bidi, etc.

[2]: https://github.com/mozilla/servo/pull/1734

[3]: https://github.com/mozilla/servo/pull/1681

[Simon Sapin]

On 25/02/2014 18:02, Patrick Walton wrote:
> So I agree that we must not make architectural decisions that make it
> difficult to implement Web features. But I do think that Web compat is
> useful, because it helps us to identify the parallel hazards and know
> where we stand in terms of the Web of today, which helps us answer the
> big questions that Servo set out to answer.

Yes, Web compat is important. But Acid3 in particular is a poor proxy
for Web compat, IMO.

--
Simon Sapin

[John Jensen]

> From my point of view, especially with my work lately, one of the most
> important questions that Servo needs to answer is "how much parallelism
> can we gain on real-world Web sites?" To that end, Web site
> compatibility (which Acid2 is a rough proxy for a large subset of, if
> used sensibly [1]) is a useful thing to work on. Otherwise, we don't
> know whether the parallel gains that we are getting will scale up to
> real-world usage, or whether we're only getting them because we don't
> implement the difficult features.

Is it possible to programmatically identify CSS rules or page layouts which would benefit from the parallelism that Servo potentially offers? For the CSS un/prefixing debate I/we had some success in deploying web crawlers that parsed and summarized millions of CSS rules from thousands of sites. Could something similar be of use for quantifying the lowest hanging parallelism fruit for Servo?

Said another way, is it possible to create a targeted benchmark for Servo compatibility and performance?

John


--
John Jensen | Director, Metrics | Mozilla Corporation

[Jack Moffitt]

> Is it possible to programmatically identify CSS rules or page layouts which would benefit from the parallelism that Servo potentially offers? For the CSS un/prefixing debate I/we had some success in deploying web crawlers that parsed and summarized millions of CSS rules from thousands of sites. Could something similar be of use for quantifying the lowest hanging parallelism fruit for Servo?
>
> Said another way, is it possible to create a targeted benchmark for Servo compatibility and performance?

I think the answer is yes, but I would be interested in talking about
this more. We've done some preliminary work discussing and estimating
this for specific sites, but it sounds like you are much farther down
this road with tooling. We'd like to leverage what you've already done
if possible.

jack.

[Boris Zbarsky]

On 2/25/14 1:02 PM, Patrick Walton wrote:
> To that end, Web site compatibility (which Acid2 is a rough proxy for a large subset of, if
> used sensibly [1])

Acid3 is a _much_ worse proxy here than Acid2, even with your sensible
use caveat, fwiw.

-Boris

[Robert O'Callahan]

On Wed, Feb 26, 2014 at 7:02 AM, Patrick Walton <pcwa...@mozilla.com>wrote:

> So I agree that we must not make architectural decisions that make it
> difficult to implement Web features. But I do think that Web compat is
> useful, because it helps us to identify the parallel hazards and know where
> we stand in terms of the Web of today, which helps us answer the big
> questions that Servo set out to answer.
>

That's a good point, but Acid3, and to a lesser extent Acid2, are about
testing edge cases and the presence of obscure features. I don't think they
tell you anything significant about parallelism in the mass of pages on the
Web. No single page can, but Acid3 is probably even worse than picking a
page at random. You'd be much better off picking, say, a Wikipedia page
(which I know you've done!) or the HTML5 single-page spec.

Otherwise, we don't know whether the parallel gains that we are getting
> will scale up to real-world usage, or whether we're only getting them
> because we don't implement the difficult features.
>

That's a good point too, but the problem is that key parts of the Web like
GMail, Youtube, Facebook etc require so much work for full support that I
don't think we draw broad conclusions until very far in the future, when it
will be far too difficult to make architectural changes. If you must
discover those conclusions earlier then we should probably piggyback some
analysis on an existing browser engine instead of doing it in Servo.

The good news is that I think CSS layout is by far the largest piece of the
Web where we have implicit parallelism that is subject to unpredictable
hazards. Other big chunks of work, like almost all DOM APIs, are either
obviously not parallelizable or obviously parallelizable.

[Patrick Walton]

On 2/25/14 3:08 PM, Robert O'Callahan wrote:
> That's a good point, but Acid3, and to a lesser extent Acid2, are about
> testing edge cases and the presence of obscure features. I don't think
> they tell you anything significant about parallelism in the mass of
> pages on the Web. No single page can, but Acid3 is probably even worse
> than picking a page at random. You'd be much better off picking, say, a
> Wikipedia page (which I know you've done!) or the HTML5 single-page spec.

So I agree about Acid3 in particular not being particularly interesting
for parallelism. (Well, except maybe the SVG stuff, but I don't think
that the real-world risk if we have to make SVG slow impacts Servo too
much.)

Wikipedia is actually a bit tricky to use as a test case in my
experience, because it makes heavy use of floats the way they were
designed to be used--for floating objects--and as a result achieves poor
parallelism right now. (Mobile Wikipedia is much faster, because it
doesn't use floats.) The Alexa Top 50 pages that I've managed to be able
to test actually generally achieve significantly better parallelism than
Wikipedia does. This is why I think it's important to have some breadth
in our tests.

> That's a good point too, but the problem is that key parts of the Web
> like GMail, Youtube, Facebook etc require so much work for full support
> that I don't think we draw broad conclusions until very far in the
> future, when it will be far too difficult to make architectural changes.
> If you must discover those conclusions earlier then we should probably
> piggyback some analysis on an existing browser engine instead of doing
> it in Servo.
>
> The good news is that I think CSS layout is by far the largest piece of
> the Web where we have implicit parallelism that is subject to
> unpredictable hazards. Other big chunks of work, like almost all DOM
> APIs, are either obviously not parallelizable or obviously parallelizable.

Right, I'm more interested in the legacy CSS layout stuff that's so
prevalent that we can't realistically push back on its use. You do have
a good point about it being hard to test things like Gmail or Facebook
due to sheer number of dynamic HTML features they use, but maybe we can
leverage existing engines and produce static versions of their pages to
test in Servo. (I actually already did this to test YouTube.) Based on
my (limited) amount of experience trying to guess how much parallelism
we'll gain by performing analyses in Gecko (in the previous thread about
measuring parallel hazards) I think it's difficult to perform these
analyses directly in existing engines without involving our actual
implementation in Servo somehow. The chaotic nature of the parallel
scheduling with work stealing makes Servo's parallel layout difficult to
model and simulate accurately.

Patrick