Experimental trybots for WPT tests in Blink

[Kristijan Burnik]

Hi to all in Chrome Infra and Blink Dev! :-)

I'm working on a new feature for running Web Platform Tests as part of the Blink Layout Test infrastructure. http://crrev.com/1154373005

DD: https://goo.gl/iXUaZd

The feature is enabled by adding the flag "--enable-wptserve" to the run-webkit-tests.

At this point, there are still some adjustments required for the tests to be able to run properly in this mode.

I am looking forward to having some experimental trybots for running those tests with the feature enabled.

I would also like to estimate the overall performance impact once we start pulling in more WPT tests which are currently being suppressed.

--

What would be the proper way for me to play around with the experimental try jobs?

And how would I get access to those?

I would like some more guidance and help with this, preferably in the CEST time zone.

--

Kristijan Burnik

Software Engineering Intern

bur...@google.com

Google Germany GmbH

Dienerstraße 12

80331 München

Geschäftsführer: Graham Law, Christine Elizabeth Flores

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

[Joshua Bell]

Sync up with joelo@ (our shiny new and awesome blink-infra person!) who can assist with making the requests. (He's Americas-Pacific TZ, though.) 

[Joel Ong]

Hey Kristijan!

I'd be more than happy to help you get some experiments going. Let's sync up in greater detail over IRC as there are lots of things we should consider before running these on the bots.

By the way, huge thanks for getting wptserve up and running! This is a big leap in the right direction -- I look forward to being able to run all of the web platform tests.

[Paweł Hajdan]

bcc:chrome-infra, +infra-dev ; please don't mix public and internal MLs

+akuegel,sergiyb,tandrii

Good news is the Commit Queue (CQ) team is in Europe, and I've added remaining CQ folks to this thread.

I'd recommend you to use the chromium recipe and add an entry for FYI bot running tests with any flags you need and a corresponding trybot in chromium_trybot recipe. Then we can add that trybot to an optional slave pool.

Paweł

--

Paweł Hajdan, Jr. |

Software Engineer |

pha...@google.com

Google Poland sp. z o.o.

ul. Emilii Plater 53, 00-113 Warszawa

Sąd Rejonowy dla m. st. Warszawy w Warszawie,

XII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 0000240611.

NIP: 525-23-44-078, Kapitał zakładowy Spółki 400 000, 00 zł.

[Philip Jägenstedt]

I can't comment on the design doc, so:

"For a particular test run via run-webkit-tests, we also closely mimic what’s being done with the apache server. We check if tests require the server (i.e. are stored in a particular directory having a substring indicating this requirement - e.g. /http/ or /web-platform-tests/) and if so we spin up the appropriate server(s)."

Is there any ambition to distinguish which tests in wpt actually need serve.py and not? AFAIK there is no reliable way to do so currently, but even in wpt most tests don't actually need the server and it might be worth checking if running those locally would affect the total test time or not.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Kristijan Burnik]

If you think it's worth checking for the requirement I would suggest to run the tests without the --enable-wptserve flag and then with the flag and see which tests fail without, but do pass with the flag. Although, there might be false positives/negatives as we are still adjusting the expectations for the wptserve version.

Otherwise we would be statically checking if a test requests a .py script as a resource, or uses subs/pipes, etc. Which might not prove as easy...

[Philip Jägenstedt]

Right, figuring this out statically seems tricky. I'm not volunteering, but if this seems at all likely to be an issue, investigating if there's actually any difference in throughput when running via wptserve and the file system would be the first step.

In any event, I'm very happy that there's some progress in this area!

[Dirk Pranke]

It seems hard to imagine that running all of the tests through wptserve won't always be at least somewhat

slower, but it's unclear whether it will be enough slower to matter. Even if it is, I suspect we'll be better off

putting effort into making other aspects of the tests faster (speeding up individual tests, doing better

sharding, distributing across machines, etc.) since, as you've both noted, it would be hard to tell which 

tests require a server and which don't.

I suspect that if we do reach the point where it's clear things are so much slower when run through a server that

it's a big deal, we'll have to work out an approach with the other w3c members to segregate tests by directory

or filename or manifest or something, much like we do w/ Blink tests.

-- Dirk