introduce blink_wpt_tests

[Weizhong Xia]

Dear blink devs

As some of you may already know, I am working to split blink_web_tests into two test suites: blink_web_tests for legacy tests and blink_wpt_tests for wpt tests(crbug/1299834). We now have reached the first point to move all baselines to a central place (CL). Some of the changes you may need to be aware of:

1. We renamed web_tests/platform to web_tests/baselines.

2. All the generic baselines are now put under web_tests/baselines/generic. (reference files are allowed at both baselines/generic or in the same folder of the test itself.)

I planned to do another round of rebaseline then land this CL the coming Saturday, to reduce the chance of merge conflict.

After that, we will have another CL to actually break blink_web_tests into two. This should have no impact on your local experiences. The only difference is that we are now running legacy tests and wpt tests at different steps, so you need to get log/results at different places. 

At this point blink_wpt_tests will continue to run with run_web_tests.py. We have a plan to switch it to use wptrunner, targeting the middle of next year. After that we will have tools, web_tests, wpt_tests under blink, and we will use metadata as test expectation for wpt_tests. More details to come.

Thanks for your attention. If you have any suggestion/comments, feel free to reply here, or put that in the crbug or CL. I definitely want to get more input from you folks.

Thanks, Weizhong

[Zijie He]

Weizhong, only SmokeTests are running on fuchsia now by default. How would this split impact the smoke tests?

[Weizhong Xia]

We will still have two test suites for fuchsia, and filter the tests properly from SmokeTests.

For the long term plan, I need to discuss with you folks (separately) on how to run wptrunner for fuchsia. 

[Harald Alvestrand]

at the moment, wpt tests live under third_party/blink/web_tests/external/wpt (4 directories down).

What will the new path for wpt tests be, once all is moved and done?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADXrSioUQu-0wzBbQm1CG2ANTH_aPgdY1NuWS309zASAhcGjtg%40mail.gmail.com.

[Weizhong Xia]

I think it would be third_party/blink/wpt_tests/, and we will have third_party/blink/wpt_tests/wpt_internal for internal tests. I would still need to check if such structure could cause problems to the importer/exporter.

[Christian Biesinger]

Will we now/later need to start using wptrunner instead of
run_web_tests.py to run the tests locally?

(BTW, what's the motivation for this change?)

Christian

[Weizhong Xia]

The goal is to use wptrunner to run wpt tests. rwt will still be used to run legacy tests. Do you imply you prefer to use rwt for both wpt tests and legacy tests? I would love to hear your reasoning about that.

There are several business motivations behind that. To be able to run tests on Android/iOS and reduce costs to developers are two of those.

[Christian Biesinger]

On Tue, Apr 26, 2022 at 2:32 PM Weizhong Xia <weiz...@google.com> wrote:
>
> The goal is to use wptrunner to run wpt tests. rwt will still be used to run legacy tests. Do you imply you prefer to use rwt for both wpt tests and legacy tests? I would love to hear your reasoning about that.

No, I just wanted to know whether I will need to switch my workflow.

> There are several business motivations behind that. To be able to run tests on Android/iOS and reduce costs to developers are two of those.

Thanks. (How does this reduce costs to developers?)

Christian

[Weizhong Xia]

Hi Folks

A follow up of this announcement. I was finally able to land this change yesterday, with a small change. We later decided to keep the platform, to reduce the size of the CL. So now all the baselines are under web_tests/platform, and generic baselines are under web_tests/platform/generic.

If you have any ongoing CL that changes baselines, pls make sure the baselines are at the correct place. Feel free to ping me if you have any questions.

thanks, Weizhong

[Weizhong Xia]

Hi Folks

Another update on this work. Today I landed the CL to run blink_wpt_tests as a different step on swarming bots. We will need to find test results at their respective steps.

Feel free to ping me if you have any questions.

thanks, Weizhong