PSA: migration to Wptrunner has started

[Weizhong Xia]

TLDR: chrome_wpt_tests and headless_shell_wpt_tests have been added to Chromium CI/CQ, and migration to Wptrunner has started.

Hello there

For quite some time web platform developers have wished to have a uniform way to run web platform tests, which is to converge to Wptrunner. Blink Engprod team has been working on this for quite some time, and now we are happy to announce that the infra is ready and we can now start to migrate the WPTs to run with Wptrunner, or run_wpt_tests.py, which is a wrapper built on top of Wptrunner and aims to make it easier to use Wptrunner in chromium.

Migration Plan

To migrate a directory to Wptrunner, you could simply add the directory to //third_party/blink/web_tests/TestLists/headless_shell.filter/, and the tests in that directory will start to run with Wptrunner + Headless shell on Chromium CI/CQ. Our plan is to fully migrate to Wptrunner by the end of the year. By that time we will have three test suites for web tests, as listed in the table below:

Suite Name	Browser Under Test	Harness	Tests Run
blink_web_tests	Content Shell	run_web_tests.py	Legacy web tests and wpt_internal
chrome_wpt_tests	Chrome new headless	run_wpt_tests.py	WPTs under external/wpt that depends on //chrome layer
headless_shell_wpt_tests	Headless Shell	run_wpt_tests.py	All other WPTs under external/wpt

How could you help?

The migration will be done directory by directory, and there are 2000+ regressions on each platform. We will start to create crbugs with the label “wptrunner-migration” for those regressions. Please watch out for the crbugs and help fix them so that we can finish the migration on time.

If you need any support during this process, the document is here. Also please feel free to reach out to weizhong@ or jonathanjlee@ if you want to have a chat.

Thank you!

Weizhong on behalf of Blink Engprod

--

Empower web platform community with a single test runner that is fast, stable and easy to use.

[PhistucK]

Sounds like headless shell will be used for the majority of the tests, but that is not testing the full browser, but just a headless shell which goes through different code paths...

I realise that previously it was probably run using content shell or something like that which is also not as representative, are there any plans to actually use the full browser for running tests?

If it is too slow, well... File bugs for making it faster ;)

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSirCEhXLO%3Dz%3DnhwCpjkfcNZ0Q8Ox-gnOBytUURp8OGReNg%40mail.gmail.com.

[Weizhong Xia]

Right, performance is the major reason chrome new headless will not be used for the majority of WPTs. One mitigation plan is to run all WPTs on CI with chrome new headless, where it is less sensitive to the total test time.