web-platform-tests that fail only in Chrome (from wpt.fyi data)

75 views
Skip to first unread message

Philip Jägenstedt

unread,
Oct 11, 2018, 5:13:36 PM10/11/18
to blin...@chromium.org
Hi all,

I've taken a look at tests that fail only in one browser on wpt.fyi and have sent reports on Safari to webkit-dev and Firefox to dev-platform. This is the same report for Chrome:

The reason that Chrome-only failures are more interesting than the average test failures is that for those that really can affect web developers, here fixing Chrome will by itself remove the trap, making the platform more predictable for web developers. Usually, that takes quite a bit more work on specs, tests, and working with other vendors.

Skimming the list, there's a lot of css/, html/ and network-related things in various directories.

CALL TO ACTION: if you know any of these areas, take a look and see if the failures represent a real problem, and if so file bugs pointing to the failing tests, or find the relevant existing bugs.

I dug into one, eventually finding an existing bug:

Thanks!

P.S. The ability to link test failures to wpt or browser bugs would of course be great to avoid repeating work here, but unfortunately is a bit of work. Hopefully there won't be much toe-stepping.

Yoichi Osato

unread,
Oct 11, 2018, 9:29:51 PM10/11/18
to Philip Jägenstedt, blin...@chromium.org
Could you create a tracking crbug for that?

2018年10月12日(金) 6:13 Philip Jägenstedt <foo...@chromium.org>:
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdoga78Pain4phhja1iXmCDOoxSt1_28%3DSCDzWSDE46Uw%40mail.gmail.com.

Philip Jägenstedt

unread,
Oct 12, 2018, 6:00:32 AM10/12/18
to Yoichi Osato, blink-dev
Sure thing, let's track these!

Two years ago I created https://crbug.com/651572 based on failures in Chrome that pass in Edge and Firefox. I've gone through what remains there and found that most of what remains was also passing in Safari, so Chrome-only failures:

That's a personal hotlist, so presumably others can't edit it. Perhaps a label would be better. Would "WPT-Chrome-Fail-Edge-Firefox-Safari-Pass" work, is there a catchier name we could use?

Aleks Totic

unread,
Oct 12, 2018, 8:22:09 PM10/12/18
to Philip Jägenstedt, blink-dev
Just took a quick look, to see if there are any quick layout fixes we
can do. Many of chrome failures are passing on the tip. You might want
to prune the list against the ToT.

passing:
/css/css-contain/contain-size-002.html
/css/css-contain/contain-size-003.html
/css/css-flexbox/css-flexbox-row-reverse-wrap-reverse.html
......
/css/css-text/i18n/ja/css-text-line-break-ja-cj-strict.html
.......\\

Philip Jägenstedt

unread,
Oct 16, 2018, 6:17:01 AM10/16/18
to Aleks Totic, Koji Ishii, blink-dev
Thanks for taking a look, Aleks!

I've run the reftests listed in my report locally and produced output that can be used with Mozilla's reftest-analyzer:

A screenshot of what that looks like:
image.png

Tip: use keyboard shortcuts 1, 2, d, p and n to quickly move through results.

As you can see, subtle differences in text rendering account for the failure, including contain-size-002.html and contain-size-003.html. css-flexbox-row-reverse-wrap-reverse.html actually passes for me locally, even with `./wpt run --verify` to run it many times. It looks like it's been flaky on wpt.fyi, I've filed https://github.com/web-platform-tests/wpt/issues/13539.

Here are the Chrome-only reftest failures that still seem worth investigating:
/css/CSS2/box-display/block-in-inline-001.xht and -002.xht
/css/CSS2/box-display/box-generation-001.xht and -002.xht
/css/CSS2/linebox/inline-formatting-context-002.xht through -007.xht
/css/CSS2/normal-flow/block-in-inline-empty-001.xht and -004.xht
/css/CSS2/normal-flow/block-in-inline-insert-001e.xht
/css/CSS2/normal-flow/block-in-inline-insert-001h.xht
/css/CSS2/normal-flow/block-in-inline-insert-002e.xht
/css/CSS2/normal-flow/max-width-applies-to-013.xht
/css/css-fonts/font-features-across-space-1.html and -3.html
/css/css-transforms/transform3d-perspective-003.html through -005.html
/css/css-transforms/transform3d-sorting-004.html
/css/css-variables/variable-generated-content-dynamic-001.html
/svg/extensibility/foreignObject/foreign-object-size.html

So, it is quite apparent that a bit of infra work is still needed to capture screenshots, and then to keep track of which Chrome-only failures we have already triaged.

Generating a new report today I also saw a lot of new Chrome-only failures show up and that explained by Widespread timeouts in Edge 17 (an infra issue), i.e. some random set of tests time out for each run.

Also, +Koji Ishii looked at failures in css/css-text/i18n/, and those are because of https://github.com/web-platform-tests/wpt/issues/13192, i.e., the tests don't depend on webfonts in quite the right way.

Outside of that I think it would still be valuable for folks to look through the original report, but I hope to have something a bit less noisy for the next round.

Philip Jägenstedt

unread,
Oct 16, 2018, 6:23:10 AM10/16/18
to Aleks Totic, Koji Ishii, blink-dev
Oh, the tracking issue for storing reftest screenshots is https://github.com/web-platform-tests/wpt.fyi/issues/57.

Philip Jägenstedt

unread,
Oct 17, 2018, 10:12:12 AM10/17/18
to Yoichi Osato, blink-dev
As suggested by Yoichi-san, I've created a tracking bug that also adds some more context:
https://bugs.chromium.org/p/chromium/issues/detail?id=896242

Please link bugs there :)

Philip Jägenstedt

unread,
Oct 17, 2018, 11:33:49 AM10/17/18
to Aleks Totic, Koji Ishii, blink-dev
Over on dev-platform, a happy accident was discovered which means we already have the ability to view results in reftest-analyzer. It's not integrated into wpt.fyi at all, but here are all the 10 shards from the latest Chrome Dev run on Taskcluster:
Working out which shard a test of interest is in will require opening them all and searching the page, I'm afraid.

Taking one failure at random, /css/CSS2/floats-clear/float-applies-to-008.xht, here's what I found in wpt-chrome-dev-reftest-9:
Screen Shot 2018-10-17 at 5.31.40 PM.png
So, uninteresting differences in font rendering.

Hopefully reftest-analyzer can be of use to investigate surprising failures. Please let me know if there are failures that have no explanation still.

Philip Jägenstedt

unread,
Oct 19, 2018, 8:50:53 AM10/19/18
to Aleks Totic, Koji Ishii, blink-dev
Following suggestions by Boris over on dev-platform, here's also a list of Chrome-only errors, i.e. neither pass nor fail, where everyone else passes:

It probably makes most sense to roll this into a single list to triage, but I've produced it as a delta in case somebody has already looked through the first list. The biggest chunk of this seems to be because of the user gesture requirement for media elements, which is tracked here:
Reply all
Reply to author
Forward
0 new messages