Must All Layout Tests Pass Locally?

PhistucK

unread,

Mar 13, 2018, 2:11:45 PM3/13/18

to blink-dev

I am using Janitor (woo hoo!) for (a trial of) finally developing Chromium (welcome, me) instead of only bothering people in the group without actually contributing code. ;)

It looks like it is some sort of an Ubuntu distribution, but without the usual graphical user interface (it uses "Fluxbox", if it means anything to you), I guess and other stuff.

> uname -a

Linux bb90bda13377 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

First, I cannot run Chromium without disabling the sandbox, because it says my kernel is not new enough and I do not have the SIUD sandbox (I guess that is a question for chromium-dev, but you are welcome to help anyway ;)).

But the more important thing (because who needs a sandbox anyway, right?), is that not all of the layout tests pass. About 100 - 300 did not pass (it was a month ago, so I am not sure about the amount).

(Disclaimer - I ran it without installing Ahem, in case it is needed. I just installed Ahem and I am re-running the layout tests, but I am already seeing some fails, like due to a GPU crash).

(I know I also might have fallen on an imperfect revision, of course, but let us assume this is not the case)

Is that normal, or are all of the layout tests expected to pass locally?

Do you run all of the layout tests locally when changing Blink?

Also, I had to define $USER or $USERNAME (I do not remember which, so I defined both of them), otherwise the tests would not run (I found my user name using whoami). Is this normal?

Yeah, I am not a Linux (or macOS) user, so I am winging it...

Thank you!

☆PhistucK

Ian Clelland

unread,

Mar 13, 2018, 2:45:42 PM3/13/18

to PhistucK, blink-dev

On Tue, Mar 13, 2018 at 6:11 PM PhistucK <phis...@gmail.com> wrote:

I am using Janitor (woo hoo!) for (a trial of) finally developing Chromium (welcome, me) instead of only bothering people in the group without actually contributing code. ;)

It looks like it is some sort of an Ubuntu distribution, but without the usual graphical user interface (it uses "Fluxbox", if it means anything to you), I guess and other stuff.
> uname -a
Linux bb90bda13377 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

First, I cannot run Chromium without disabling the sandbox, because it says my kernel is not new enough and I do not have the SIUD sandbox (I guess that is a question for chromium-dev, but you are welcome to help anyway ;)).

But the more important thing (because who needs a sandbox anyway, right?), is that not all of the layout tests pass. About 100 - 300 did not pass (it was a month ago, so I am not sure about the amount).
(Disclaimer - I ran it without installing Ahem, in case it is needed. I just installed Ahem and I am re-running the layout tests, but I am already seeing some fails, like due to a GPU crash).

Ahem is definitely needed for some of the layout tests; it's a font specifically designed to provide pixel-perfect consistency when rendered in any environment, and the -expected.png files are only going to match the results of your tests if you have that font installed.

(I know I also might have fallen on an imperfect revision, of course, but let us assume this is not the case)
Is that normal, or are all of the layout tests expected to pass locally?

I believe that all of the tests (except for those which are specifically marked as failing in the various TestExpectations files, but the test runner automatically takes that into account) *should* pass locally. If they don't, the test may be flaky, or may depend on some configuration assumption, like the installed font list.

Do you run all of the layout tests locally when changing Blink?

Definitely not :) That would take too long for me, even on my Google-provided workstation. I'll definitely run the tests that I believe are affected by my changes, and if I'm just trying to get a single test to pass, I'll usually run all of the other tests in that directory to catch regressions early, but mostly I'll let the trybots handle running *all* of them, while I try to work on something else.

Also, I had to define $USER or $USERNAME (I do not remember which, so I defined both of them), otherwise the tests would not run (I found my user name using whoami). Is this normal?

Yeah, I am not a Linux (or macOS) user, so I am winging it...

Thank you!

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_LdfuA1Vnm1AYFbSuO9WOYaOA9HWGB61hqFaeiv8ycVpQ%40mail.gmail.com.

Aleks Totic

unread,

Mar 13, 2018, 2:56:39 PM3/13/18

to PhistucK, blink-dev

I run full layout test suite multiple times daily (takes ~10m), and most days, most tests pass.

You will need Ahem. I thought there were other fonts needed too, ./build/install-build-deps.sh installs them?

I am unable to run the test suite without failures when chromoting. For some reason, bunch of media tests fail with timeouts. If you do not have a GUI, your environment might be closer to chromoting one, GPU crashes sound familiar.

This is the command line I use for fast test runs:

third_party/WebKit/Tools/Scripts/run-webkit-tests --target=Optimized $TEST_FLAGS --skip-timeouts $SLOW_TESTS --order=none --num-retries=1

out/Optimized is optimized Chromium compile, with dchecks on.

TEST_FLAGS is a skip list of tests that are really annoying (long timeouts on my machine)

--skip-timeouts skips tests that are expected to timeout. Nobody has time for that.

--order=none to minimize flakiness

--num-retries=1 shows me the results sooner, and catches most tests that pass on retry.

Aleks

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

PhistucK

unread,

Mar 13, 2018, 3:48:13 PM3/13/18

to Aleks Totic, blink-dev

First, thank you, both of you!

Interesting information. I am running through a Cloud9 terminal, so no graphical user interface at that point, but I can run in from within VNC which provides a desktop environment. Perhaps I will have better luck with more passes this way.

I will look into those flags, though I do not intend to run them a lot (so I do not mind not skipping timeouts), I just wanted to make sure I have a successful base build in terms of layout tests before I even think about changing any code.

It seems like install-build-deps.sh did not install Ahem, unfortunately. Placing it under ~/.fonts made Chromium display the font, so I guess it was not installed before.

Is it expected that out/Default/layout-test-results/results.html is not created (I have not finished re-running yet, but I do not think it was there after the first run)?

And is it a worthwhile file to look at (nicer than the terminal output, with actual comparisons and similar, for example), or is it false hope?

Once the re-run is over, I will try to run it from within VNC and report the results

Some preliminary supposedly unexpected test failures from the Cloud9 re-run -

[522/64579] virtual/exotic-color-space/images/webp-color-profile-lossy.html failed unexpectedly (gpu crashed)

[872/64579] virtual/incremental-shadow-dom/externa...n-shadow-trees/html-forms/test-002.html passed unexpectedly

[1016/64579] virtual/gpu-rasterization/images/optimize-contrast-canvas.html failed unexpectedly (gpu crashed)

[1140/64579] virtual/incremental-shadow-do...ontroller-recursion-crash.html failed unexpectedly (test timed out)

[1254/64579] virtual/gpu/fast/canvas/pixelated-resize.html failed unexpectedly (test timed out)

[1312/64579] http/tests/devtools/resource-parameters-ipv6.js failed unexpectedly (test timed out)

[1631/64579] virtual/gpu/fast...r-in-worker.html failed unexpectedly (test timed out, -expected.txt was missing)

[2055/64579] virtual/gpu/fast/canvas/canvas-filter-removed.html failed unexpectedly (renderer crashed)

[2092/64579] virtual/gpu/fast/canvas/canvas-lost-gpu-context.html failed unexpectedly (test timed out)

[2311/64579] http/tests/devtools/console/command-line-api.js failed unexpectedly (test timed out)

[3854/64579] virtual/layout_ng/fast/inline/break-between-nobr.html passed unexpectedly

[4167/64579] virtual/mojo-blobs/external/wpt/FileAPI/url/sandboxed-iframe.html failed unexpectedly (text diff)

[4349/64579] virtual/mojo-blobs/external/w...xedDB/interleaved-cursors.html failed unexpectedly (test timed out)

[4610/64579] virtual/mojo-blobs/external/w...alScope/postmessage.https.html failed unexpectedly (test timed out)

[4622/64579] virtual/mojo-blobs/external/w...istration-attribute.https.html failed unexpectedly (asserts failed)

[4684/64579] virtual/mojo-blobs/external/wpt/webmessaging/Channel_postMessage_Blob.htm passed unexpectedly

[5054/64579] virtual/mouseevent_fractional/fa...d-close-different-microtask.html failed unexpectedly (text diff)

[5172/64579] virtual/mouseevent_fractional...hr-timestamp/input-events.html failed unexpectedly (asserts failed)

[5227/64579] virtual/mojo-blobs/external/w...navigation-redirect.https.html failed unexpectedly (test timed out)

[5255/64579] virtual/mojo-blobs/e...ctivation.https.html failed unexpectedly (content_shell crashed [pid=17710])

[5268/64579] virtual/mouseevent_fractional...-in-zoom-and-scroll-right.html failed unexpectedly (test timed out)

[5278/64579] virtual/mouseevent_fractional...touchstart-in-slop-region.html failed unexpectedly (test timed out)

[5324/64579] virtual/mouseevent_fractional/fa.../mouse-pointer-on-scrollbar.html failed unexpectedly (text diff)

[5358/64579] http/tests/devtools/elements/eve...framework-with-service-worker.js failed unexpectedly (text diff)

[5360/64579] virtual/mouseevent_fractional...nter-event-in-slop-region.html failed unexpectedly (test timed out)

[5361/64579] virtual/mouseevent_fractional...coords-in-zoom-and-scroll.html failed unexpectedly (test timed out)

[5398/64579] virtual/mouseevent_fractional/fast/events/hit-test-counts.html failed unexpectedly (gpu crashed)

[5404/64579] virtual/mouseevent_fractional...ents/tap-on-scaled-screen.html failed unexpectedly (test timed out)

[5590/64579] virtual/mouseevent_fractional/fast/eve...ainthread-touchpad-fling-latching.html passed unexpectedly

[5624/64579] virtual/mouseevent_fractional/fast/eve...heel-scroll-latching-on-scrollbar.html passed unexpectedly

[5841/64579] virtual/new-remote-playback-p...ring-doesnt-show-controls.html failed unexpectedly (asserts failed)

[6097/64579] virtual/mouseevent_fractional.../events/no-fake-mousemove.html failed unexpectedly (test timed out)

[6120/64579] virtual/mouseevent_fractional/f...-key-context-menu-document.html failed unexpectedly (gpu crashed)

[6292/64579] virtual/mouseevent_fractional/fa...useenter-mouseleave-on-drag.html failed unexpectedly (text diff)

[7844/64579] virtual/outofblink-cors/exter...navigation-redirect.https.html failed unexpectedly (test timed out)

[7866/64579] virtual/outofblink-cors/external/wpt/s...rker/fetch-request-redirect.https.html passed unexpectedly

[8102/64579] virtual/outofblink-cors/exte...egister-new-script.https.html failed unexpectedly (renderer crashed)

[8132/64579] virtual/rootlayerscrolls/fast/sc...ubpixel-overflow-mouse-drag.html failed unexpectedly (text diff)

[8147/64579] virtual/rootlayerscrolls/fast/scrolling/scrollbar-tickmarks-hittest.html passed unexpectedly

[8149/64579] virtual/rootlayerscrolls/fast/scrolling/scrollbar-prevent-default.html passed unexpectedly

[8627/64579] virtual/scroll_customization/...olling/hashtag-autoscroll.html failed unexpectedly (asserts failed)

[8704/64579] virtual/service-worker-script...navigation-redirect.https.html failed unexpectedly (test timed out)

[8713/64579] virtual/service-worker-script...alScope/postmessage.https.html failed unexpectedly (test timed out)

[9328/64579] virtual/service-worker-script...-worker/fetch-event.https.html failed unexpectedly (test timed out)

[9426/64579] virtual/service-worker-script...e-worker/activation.https.html failed unexpectedly (asserts failed)

[10655/64579] virtual/spv175/paint/images/im...ackgrounds-not-antialiased.html failed unexpectedly (gpu crashed)

[10656/64579] virtual/spv175/fragm...wspan-alignment.html failed unexpectedly (content_shell crashed [pid=4298])

[11629/64579] http/tests/devtools/profiler/heap-snapshot-event-listeners.js failed unexpectedly (test timed out)

[11968/64579] virtual/stable/webexposed/global-interface-listing.html failed unexpectedly (test timed out)

[12889/64579] http/tests/devtools/service-workers/service-workers-view.js failed unexpectedly (text diff)

[12906/64579] http/tests/devtools/service-workers/service-worker-manager.js failed unexpectedly (text diff)

[12919/64579] virtual/threaded/animations/i...-dasharray-interpolation.html failed unexpectedly (test timed out)

[13320/64579] accessibility/table-header-column-row.html failed unexpectedly (test timed out)

[13493/64579] accessibility/element-role-mapping-normal.html failed unexpectedly (test timed out)

[14652/64579] animations/interpolation/svg-...-dasharray-interpolation.html failed unexpectedly (test timed out)

☆PhistucK

Dirk Pranke

unread,

Mar 13, 2018, 4:25:36 PM3/13/18

to PhistucK, Aleks Totic, blink-dev, Thomas Anderson

We've been landing changes to pull all of the fonts directly from the checkout rather than installing them, but I'm not sure what the current state of that is.

+thomasanderson could say more.

-- Dirk

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BoKDFW8SpQV45y9ajgPG7mL9VAaZAsUbG_cao8E7CsGw%40mail.gmail.com.

Message has been deleted

Robert Ma

unread,

Mar 13, 2018, 5:31:33 PM3/13/18

to blink-dev, phis...@gmail.com, ato...@google.com, thomasa...@chromium.org

I've worked on the layout test infra for a while. The ideal state we want to achieve (and hopefully the current state) is that all external dependencies are managed via gclient hooks (especially content_shell_fonts) & install-build-deps.sh.

Contrary to many people would think, Ahem is not required to be installed on your system to run layout tests. Layout tests use the content_shell test runner instead of actual Chrome. And we directly inject Ahem to content_shell on all three platforms (e.g. Linux). You may need Ahem in system if you want to debug layout tests or WPT in Chrome/Chromium instead of content_shell.

If you think there is any dependency missing, it's a bug.

However, zero external dependencies don't necessarily mean all the tests would pass in your local dev environment. We have many layout tests that depend on system configurations for which we need to check in system-dependent baselines (e.g. "pixel tests" compare screenshots of rendered content against baselines). And we only check in baselines for a very limited set of configurations (those in our testing infra, i.e. waterfall and CQ). Besides, some tests are resource-intense and might time out on slower machines.

Is it expected that out/Default/layout-test-results/results.html is not created (I have not finished re-running yet, but I do not think it was there after the first run)?

It will be created after the test run finishes. I'd suggest you to run smaller chunks of tests (directories that interest you) to make both the run time and the number of failures tractable.

Hope that helps.

Aleks Totic

unread,

Mar 13, 2018, 7:29:11 PM3/13/18

to Tom Anderson, Robert Ma, Thomas Anderson, blink-dev, phis...@gmail.com

It’d be awesome if they worked when chromoting. Setting DISPLAY:0 helps a bit, but still lots of failures.

On Tue, Mar 13, 2018 at 3:35 PM Tom Anderson <thomasa...@google.com> wrote:

On Tue, Mar 13, 2018 at 2:31 PM, Robert Ma <robe...@chromium.org> wrote:
I've worked on the layout test infra for a while. The ideal state we want to achieve (and hopefully the current state) is that all external dependencies are managed via gclient hooks (especially content_shell_fonts) & install-build-deps.sh.

Contrary to many people would think, Ahem is not required to be installed on your system to run layout tests. Layout tests use the content_shell test runner instead of actual Chrome. And we directly inject Ahem to content_shell on all three platforms (e.g. Linux). You may need Ahem in system if you want to debug layout tests or WPT in Chrome/Chromium instead of content_shell.

If you think there is any dependency missing, it's a bug.

However, zero external dependencies don't necessarily mean all the tests would pass in your local dev environment. We have many layout tests that depend on system configurations for which we need to check in system-dependent baselines (e.g. "pixel tests" compare screenshots of rendered content against baselines). And we only check in baselines for a very limited set of configurations (those in our testing infra, i.e. waterfall and CQ). Besides, some tests are resource-intense and might time out on slower machines.

We're trying to make it so that the tests pass on any machine, regardless of configuration [CL]. We statically link freetype, harfbuzz, and fontconfig, and we pull in all necessary fonts and set up our own fonts.conf that has a sort of "jail" to make sure we don't load any system fonts. In theory it should work on any system since we'll pretty much control everything :)

Robert Ma

unread,

Mar 13, 2018, 10:00:55 PM3/13/18

to thomasa...@google.com, blink-dev, phis...@gmail.com, Aleks Totic, thomasa...@chromium.org

On Tue, Mar 13, 2018 at 6:35 PM Tom Anderson <thomasa...@google.com> wrote:

We're trying to make it so that the tests pass on any machine, regardless of configuration [CL]. We statically link freetype, harfbuzz, and fontconfig, and we pull in all necessary fonts and set up our own fonts.conf that has a sort of "jail" to make sure we don't load any system fonts. In theory it should work on any system since we'll pretty much control everything :)

That's really awesome news! So excited to see it happening. Nice job, Tom!

Aleks Totic

unread,

Mar 19, 2018, 12:00:57 PM3/19/18

to Tom Anderson, Robert Ma, Thomas Anderson, blink-dev, PhistucK

On Tue, Mar 13, 2018 at 4:32 PM, Tom Anderson <thomasa...@google.com> wrote:

On Tue, Mar 13, 2018 at 4:28 PM, Aleks Totic <ato...@google.com> wrote:
It’d be awesome if they worked when chromoting. Setting DISPLAY:0 helps a bit, but still lots of failures.

Try running the layout tests under xvfb. Should be $ tools/xvfb.py <what you'd normally use to run layout tests>

I've tried running tests under xvfb while chromoting. 232 tests time out. Mostly media related. Is there a tracking bug for this?

Aleks

Timeout tests:

 compositing/video/video-reflection.html

 external/wpt/2dcontext/imagebitmap/createImageBitmap-origin.sub.html

 external/wpt/content-security-policy/media-src/media-src-7_1.html

 external/wpt/content-security-policy/media-src/media-src-7_2.html

 external/wpt/content-security-policy/media-src/media-src-7_3.sub.html

 external/wpt/content-security-policy/media-src/media-src-redir-bug.sub.html

 external/wpt/feature-policy/autoplay-allowed-by-feature-policy-attribute.https.sub.html

 external/wpt/feature-policy/autoplay-allowed-by-feature-policy.https.sub.html

 external/wpt/feature-policy/experimental-features/unsized-image.tentative.https.sub.html

 external/wpt/html/browsers/browsing-the-web/read-media/pageload-video.html

 external/wpt/html/semantics/embedded-content/media-elements/audio_loop_base.html

 external/wpt/html/semantics/embedded-content/media-elements/autoplay-with-broken-track.html

 external/wpt/html/semantics/embedded-content/media-elements/event_canplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_canplay_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_canplaythrough.html

 external/wpt/html/semantics/embedded-content/media-elements/event_canplaythrough_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_loadeddata.html

 external/wpt/html/semantics/embedded-content/media-elements/event_loadeddata_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_loadedmetadata.html

 external/wpt/html/semantics/embedded-content/media-elements/event_loadedmetadata_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_loadstart.html

 external/wpt/html/semantics/embedded-content/media-elements/event_loadstart_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_order_canplay_canplaythrough.html

 external/wpt/html/semantics/embedded-content/media-elements/event_order_canplay_playing.html

 external/wpt/html/semantics/embedded-content/media-elements/event_order_loadedmetadata_loadeddata.html

 external/wpt/html/semantics/embedded-content/media-elements/event_order_loadstart_progress.html

 external/wpt/html/semantics/embedded-content/media-elements/event_pause.html

 external/wpt/html/semantics/embedded-content/media-elements/event_pause_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_play.html

 external/wpt/html/semantics/embedded-content/media-elements/event_play_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_playing.html

 external/wpt/html/semantics/embedded-content/media-elements/event_playing_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_progress.html

 external/wpt/html/semantics/embedded-content/media-elements/event_progress_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/event_timeupdate.html

 external/wpt/html/semantics/embedded-content/media-elements/event_timeupdate_noautoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/networkState_during_loadstart.html

 external/wpt/html/semantics/embedded-content/media-elements/networkState_during_progress.html

 external/wpt/html/semantics/embedded-content/media-elements/paused_false_during_play.html

 external/wpt/html/semantics/embedded-content/media-elements/paused_true_during_pause.html

 external/wpt/html/semantics/embedded-content/media-elements/readyState_during_canplay.html

 external/wpt/html/semantics/embedded-content/media-elements/readyState_during_canplaythrough.html

 external/wpt/html/semantics/embedded-content/media-elements/readyState_during_loadeddata.html

 external/wpt/html/semantics/embedded-content/media-elements/readyState_during_loadedmetadata.html

 external/wpt/html/semantics/embedded-content/media-elements/readyState_during_playing.html

 external/wpt/html/semantics/embedded-content/media-elements/video_008.htm

 external/wpt/html/semantics/embedded-content/media-elements/video_loop_base.html

 external/wpt/html/semantics/embedded-content/media-elements/error-codes/error.html

 external/wpt/html/semantics/embedded-content/media-elements/interfaces/TextTrack/activeCues.html

 external/wpt/html/semantics/embedded-content/media-elements/loading-the-media-resource/autoplay-overrides-preload.html

 external/wpt/html/semantics/embedded-content/media-elements/loading-the-media-resource/load-events-networkState.html

 external/wpt/html/semantics/embedded-content/media-elements/offsets-into-the-media-resource/currentTime.html

 external/wpt/html/semantics/embedded-content/media-elements/playing-the-media-resource/pause-move-to-other-document.html

 external/wpt/html/semantics/embedded-content/media-elements/playing-the-media-resource/pause-move-within-document.html

 external/wpt/html/semantics/embedded-content/media-elements/playing-the-media-resource/pause-remove-from-document.html

 external/wpt/html/semantics/embedded-content/media-elements/ready-states/autoplay-with-slow-text-tracks.html

 external/wpt/html/semantics/embedded-content/media-elements/ready-states/autoplay.html

 external/wpt/html/semantics/embedded-content/media-elements/seeking/seek-to-currentTime.html

 external/wpt/html/semantics/embedded-content/media-elements/seeking/seek-to-max-value.htm

 external/wpt/html/semantics/embedded-content/media-elements/seeking/seek-to-negative-time.htm

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-active-cues.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-cue-negative-duration.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-cue-negative-timestamp-events.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-cue-rendering-empty-cue.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-cues-missed.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-cues-pause-on-exit.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-cues-seeking.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-cues-sorted-before-dispatch.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-delete-during-setup.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-disabled-addcue.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-disabled.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-insert-after-load.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-mode-not-changed-by-new-track.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-mode-triggers-loading.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-mode.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-remove-active-cue.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-remove-insert-ready-state.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-remove-track.html

 external/wpt/html/semantics/embedded-content/media-elements/track/track-element/track-webvtt-non-snap-to-lines.html

 external/wpt/media-source/mediasource-activesourcebuffers.html

 external/wpt/media-source/mediasource-append-buffer.html

 external/wpt/media-source/mediasource-appendwindow.html

 external/wpt/media-source/mediasource-avtracks.html

 external/wpt/media-source/mediasource-buffered.html

 external/wpt/media-source/mediasource-config-change-webm-a-bitrate.html

 external/wpt/media-source/mediasource-config-change-webm-av-audio-bitrate.html

 external/wpt/media-source/mediasource-config-change-webm-av-framesize.html

 external/wpt/media-source/mediasource-config-change-webm-av-video-bitrate.html

 external/wpt/media-source/mediasource-detach.html

 external/wpt/media-source/mediasource-duration-boundaryconditions.html

 external/wpt/media-source/mediasource-duration.html

 external/wpt/media-source/mediasource-errors.html

 external/wpt/media-source/mediasource-liveseekable.html

 external/wpt/media-source/mediasource-play-then-seek-back.html

 external/wpt/media-source/mediasource-play.html

 external/wpt/media-source/mediasource-remove.html

 external/wpt/media-source/mediasource-removesourcebuffer.html

 external/wpt/media-source/mediasource-seek-beyond-duration.html

 external/wpt/media-source/mediasource-seek-during-pending-seek.html

 external/wpt/media-source/mediasource-seekable.html

 external/wpt/media-source/mediasource-sequencemode-append-buffer.html

 external/wpt/media-source/mediasource-sourcebuffer-mode.html

 external/wpt/media-source/mediasource-sourcebuffer-trackdefaults.html

 external/wpt/media-source/mediasource-timestamp-offset.html

 external/wpt/mediacapture-fromelement/capture.html

 external/wpt/mediacapture-fromelement/creation.html

 external/wpt/mediacapture-fromelement/ended.html

 external/wpt/mediacapture-streams/MediaStreamTrackEvent-constructor.https.html

 external/wpt/mixed-content/audio-tag/http-csp/same-host-https/top-level/keep-scheme-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/audio-tag/http-csp/same-host-https/top-level/no-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/audio-tag/meta-csp/same-host-https/top-level/no-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/cross-origin-http/top-level/keep-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/cross-origin-http/top-level/no-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/cross-origin-http/top-level/swap-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/same-host-http/top-level/keep-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/same-host-http/top-level/no-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/same-host-http/top-level/swap-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/same-host-https/top-level/keep-scheme-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/audio-tag/no-opt-in/same-host-https/top-level/no-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/video-tag/http-csp/same-host-https/top-level/keep-scheme-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/video-tag/http-csp/same-host-https/top-level/no-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/video-tag/meta-csp/same-host-https/top-level/no-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/cross-origin-http/top-level/keep-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/cross-origin-http/top-level/no-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/cross-origin-http/top-level/swap-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/same-host-http/top-level/keep-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/same-host-http/top-level/no-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/same-host-http/top-level/swap-scheme-redirect/optionally-blockable/no-opt-in-allows.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/same-host-https/top-level/keep-scheme-redirect/allowed/allowed.https.html

 external/wpt/mixed-content/video-tag/no-opt-in/same-host-https/top-level/no-redirect/allowed/allowed.https.html

 external/wpt/preload/single-download-preload.html

 external/wpt/resource-timing/resource_initiator_types.html

 external/wpt/service-workers/service-worker/fetch-canvas-tainting-video-cache.https.html

 external/wpt/service-workers/service-worker/fetch-canvas-tainting-video-with-range-request.https.html

 external/wpt/service-workers/service-worker/fetch-canvas-tainting-video.https.html

 external/wpt/service-workers/service-worker/fetch-request-redirect.https.html

 external/wpt/webaudio/the-audio-api/the-analysernode-interface/test-analyser-scale.html

 external/wpt/webaudio/the-audio-api/the-analysernode-interface/test-analysernode.html

 external/wpt/webaudio/the-audio-api/the-audiobuffer-interface/idl-test.html

 external/wpt/webaudio/the-audio-api/the-audiodestinationnode-interface/idl-test.html

 external/wpt/webaudio/the-audio-api/the-audioparam-interface/idl-test.html

 external/wpt/webaudio/the-audio-api/the-audioworklet-interface/audioworklet-addmodule-resolution.https.html

 external/wpt/webaudio/the-audio-api/the-audioworklet-interface/audioworklet-messageport.https.html

 external/wpt/webaudio/the-audio-api/the-audioworklet-interface/audioworkletnode-construction.https.html

 external/wpt/webaudio/the-audio-api/the-audioworklet-interface/audioworkletprocessor-options.https.html

 external/wpt/webaudio/the-audio-api/the-audioworklet-interface/baseaudiocontext-audioworklet.https.html

 external/wpt/webaudio/the-audio-api/the-constantsourcenode-interface/constant-source-basic.html

 external/wpt/webaudio/the-audio-api/the-constantsourcenode-interface/test-constantsourcenode.html

 external/wpt/webaudio/the-audio-api/the-convolvernode-interface/convolver-setBuffer-null.html

 external/wpt/webaudio/the-audio-api/the-delaynode-interface/idl-test.html

 external/wpt/webaudio/the-audio-api/the-dynamicscompressornode-interface/dynamicscompressor-basic.html

 external/wpt/webaudio/the-audio-api/the-gainnode-interface/gain-basic.html

 external/wpt/webaudio/the-audio-api/the-gainnode-interface/idl-test.html

 external/wpt/webaudio/the-audio-api/the-iirfilternode-interface/test-iirfilternode.html

 external/wpt/webaudio/the-audio-api/the-pannernode-interface/pannernode-basic.html

 external/wpt/webaudio/the-audio-api/the-stereopanner-interface/stereopannernode-basic.html

 external/wpt/webvtt/api/VTTRegion/non-visible-cue-with-region.html

 fast/canvas/canvas-drawImage-live-video.html

 fast/events/media-focus-in-standalone-media-document.html

 fast/events/tabindex-focus-blur-all.html

 fast/mediastream/MediaStreamTrack-clone.html

 fast/mediastream/MediaStreamTrack-contentHint.html

 fast/mediastream/MediaStreamTrack-getCapabilities.html

 fast/spatial-navigation/snav-media-elements.html

 fullscreen/full-screen-iframe-allowed-video.html

 fullscreen/video-controls-timeline.html

 fullscreen/video-specified-size.html

 hdr/color-profile-video.html

 hdr/video-canvas-alpha.html

 paint/invalidation/media-audio-no-spurious-repaints.html

 paint/invalidation/video-mute-repaint.html

 paint/invalidation/video-paint-invalidation.html

 paint/invalidation/video-unmute-repaint.html

 virtual/feature-policy-permissions/external/wpt/mediacapture-streams/MediaStreamTrackEvent-constructor.https.html

 virtual/gpu/fast/canvas/canvas-drawImage-live-video.html

 virtual/modern-media-controls/media/controls/modern/doubletap-on-play-button.html

 virtual/modern-media-controls/media/controls/modern/doubletap-to-jump-backwards-at-start.html

 virtual/modern-media-controls/media/controls/modern/doubletap-to-jump-backwards.html

 virtual/modern-media-controls/media/controls/modern/doubletap-to-jump-forwards-too-short.html

 virtual/modern-media-controls/media/controls/modern/doubletap-to-jump-forwards.html

 virtual/modern-media-controls/media/controls/modern/doubletap-to-toggle-fullscreen.html

 virtual/modern-media-controls/media/controls/modern/quirks-mode-timeline-is-not-hidden.html

 virtual/modern-media-controls/media/controls/modern/scrubbing-stops-when-controls-hidden.html

 virtual/modern-media-controls/media/controls/modern/scrubbing-touch.html

 virtual/modern-media-controls/media/controls/modern/singletap-on-outside.html

 virtual/modern-media-controls/media/controls/modern/singletap-on-overlay-closes-overflow-menu.html

 virtual/modern-media-controls/media/controls/modern/singletap-on-play-button.html

 virtual/modern-media-controls/media/controls/modern/slow-doubletap.html

 virtual/modern-media-controls/media/controls/modern/tap-to-hide-controls.html

 virtual/modern-media-controls/media/controls/modern/video-tag-with-only-audio-looks-like-audio-tag.html

 virtual/mse-1mb-buffers/http/tests/media/media-source/stream_memory_tests/mediasource-appendbuffer-quota-exceeded-1mb-buffers.html

 virtual/picture-in-picture/external/wpt/feature-policy/autoplay-allowed-by-feature-policy-attribute.https.sub.html

 virtual/picture-in-picture/external/wpt/feature-policy/autoplay-allowed-by-feature-policy.https.sub.html

 virtual/picture-in-picture/external/wpt/feature-policy/experimental-features/unsized-image.tentative.https.sub.html

 virtual/picture-in-picture/media/picture-in-picture/controls/picture-in-picture-button.html

 virtual/unified-autoplay/external/wpt/feature-policy/autoplay-allowed-by-feature-policy-attribute-redirect-on-load.https.sub.html

 virtual/unified-autoplay/external/wpt/feature-policy/autoplay-allowed-by-feature-policy-attribute.https.sub.html

 virtual/unified-autoplay/external/wpt/feature-policy/autoplay-allowed-by-feature-policy.https.sub.html

 virtual/unified-autoplay/external/wpt/feature-policy/autoplay-default-feature-policy.https.sub.html

 virtual/unified-autoplay/external/wpt/feature-policy/autoplay-disabled-by-feature-policy.https.sub.html

 virtual/unified-autoplay/external/wpt/feature-policy/experimental-features/unsized-image.tentative.https.sub.html

 webaudio/audio-scheduled-source-basic.html

 webaudio/dom-exceptions.html

 webaudio/test-basic.html

 webaudio/Analyser/realtimeanalyser-basic.html

 webaudio/AudioBuffer/audiobuffer-copy-channel.html

 webaudio/AudioBuffer/audiobuffer-getChannelData.html

 webaudio/AudioBuffer/audiobuffer.html

 webaudio/AudioBufferSource/audiobuffersource-channels.html

 webaudio/AudioContext/audiocontext-close-basic.html

 webaudio/AudioContext/audiocontext-getoutputtimestamp.html

 webaudio/AudioContext/audiocontext-listener-should-not-crash.html

 webaudio/AudioContext/audiocontext-suspend-resume.html

 webaudio/AudioContext/audiocontextoptions.html

 webaudio/AudioNode/audionode-connect-method-chaining.html

 webaudio/AudioNode/audionode-disconnect-audioparam.html

 webaudio/AudioNode/audionode.html

 webaudio/AudioParam/audioparam-method-chaining.html

 webaudio/BiquadFilter/biquad-getFrequencyResponse.html

 webaudio/BiquadFilter/biquadfilternode-basic.html

 webaudio/MediaElementAudioSource/mediaelementaudiosourcenode.html

 webaudio/MediaStreamAudioDestination/mediastreamaudiodestinationnode.html

 webaudio/MediaStreamAudioSource/mediastreamaudiosourcenode.html

 webaudio/codec-tests/webm/webm-decode.html

 webaudio/constructor/mediastreamaudiodestination.html

 webaudio/constructor/mediastreamaudiosource.html

 webaudio/decodeAudioData/decode-audio-data-basic.html

 webaudio/internals/audiocontext-close.html

 webaudio/internals/audiocontext-lock-threading-race.html

 webaudio/internals/audiosummingjunction-crash.html

 webaudio/internals/mediaelementaudiosourcenode-wrapper.html

 webaudio/unit-tests/audit.html

PhistucK

unread,

Mar 19, 2018, 12:24:27 PM3/19/18

to Tom Anderson, Aleks Totic, Robert Ma, Thomas Anderson, blink-dev

testing/xvfb.py, you mean?

I briefly tried it, following your suggestion, but it did not even start running any test before it simply quit. I might try again soon, however, at the end of the non-xvfb run, only about 6 tests unexpectedly failed (and about ~200 passed unexpectedly, which is weird, I suppose, but maybe it was an intermediate commit before baselining/updating expectations...).

☆PhistucK

On Wed, Mar 14, 2018 at 1:32 AM, Tom Anderson <thomasa...@google.com> wrote:

On Tue, Mar 13, 2018 at 4:28 PM, Aleks Totic <ato...@google.com> wrote:
It’d be awesome if they worked when chromoting. Setting DISPLAY:0 helps a bit, but still lots of failures.

Try running the layout tests under xvfb. Should be $ tools/xvfb.py <what you'd normally use to run layout tests>

On Tue, Mar 13, 2018 at 3:35 PM Tom Anderson <thomasa...@google.com> wrote:

Robert Ma

unread,

Mar 19, 2018, 1:37:47 PM3/19/18

to PhistucK Productions, Tom Anderson, Aleks Totic, thomasa...@chromium.org, blink-dev

On Mon, Mar 19, 2018 at 12:24 PM PhistucK <phis...@gmail.com> wrote:

testing/xvfb.py, you mean?
I briefly tried it, following your suggestion, but it did not even start running any test before it simply quit. I might try again soon, however, at the end of the non-xvfb run, only about 6 tests unexpectedly failed (and about ~200 passed unexpectedly, which is weird, I suppose, but maybe it was an intermediate commit before baselining/updating expectations...).

Even a very recent run on the waterfall has ~50 unexpected passes. Most of them are out-of-date (need to be removed) or inaccurate (should be marked as flaky instead of failures) test expectations. Unfortunately, TestExpectations could be hard to keep track of. But ~200 might just be your local intermediate state.

And you are not seeing that 200+ timeouts Aleks mentioned?

PhistucK

unread,

Mar 19, 2018, 2:12:09 PM3/19/18

to Robert Ma, Tom Anderson, Aleks Totic, Thomas Anderson, blink-dev

There were many timeouts, but after an automatic rerun (I think a third automatic rerun), the tests passed and only 4 remained (without xvfb and without an X server, it seems). I have not investigated them -

http/tests/devtools/resource-parameters-ipv6.js - text difference, example -

15 httpVersion : "HTTP/1.1"

13 httpVersion : ""

http/tests/devtools/tracing/decode-resize.js - timeout

http/tests/misc/bad-charset-alias.html - text difference -

3 SUCCESS

3 FAIL: foobar 8859_1 ISO8859_1 8859-1 ISO_2022,locale=ja,version=0 utf 8 utf_8 8859 1 8859*1 8859:1 88591 ISO_88591 ISO-88591 ISO-88-59-1 latin-1

virtual/threaded/http/tests/devtools/tracing/decode-resize.js - crash

☆PhistucK

Aleks Totic

unread,

Mar 21, 2018, 6:24:56 PM3/21/18

to Tom Anderson, PhistucK, Robert Ma, Thomas Anderson, blink-dev

Is there a tracking bug, so I know when it is safe to try again?

On Mon, Mar 19, 2018 at 11:41 AM, Tom Anderson <thomasa...@google.com> wrote:

Those are known failures on newer Linux distros. We have a bot testing this and we hope to have them fixed soon:
https://ci.chromium.org/buildbot/chromium.fyi/Linux%20Xenial/112

Reply all

Reply to author

Forward