8% of test changes have been in wpt since Jan 17

[Philip Jägenstedt]

I wrote a script to score our "20% of Blink CLs in LayoutTests/ are in LayoutTests/external/wpt/" KR and the result is that there were 1019 test changes, of which 84 were in wpt, just above 8%. We'd get a 0.41 score based on that, but let's measure again on March 31.

(Jan 17 is when LayoutTests/imported/ was moved to LayoutTests/external/, measuring before that would complicate the script.)

[Rick Byers]

Thanks for doing the analysis!  84 CLs modifying wpt since January is a great start (plus all the WPT work chromium folks do upstream in GitHub of course)!  This is better than I feared, though obviously still not as high as we'd like.  It'll be interesting to see how this changes as we start asking more about WPT in intents...

Rick

On Wed, Mar 22, 2017 at 5:01 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

I wrote a script to score our "20% of Blink CLs in LayoutTests/ are in LayoutTests/external/wpt/" KR and the result is that there were 1019 test changes, of which 84 were in wpt, just above 8%. We'd get a 0.41 score based on that, but let's measure again on March 31.

(Jan 17 is when LayoutTests/imported/ was moved to LayoutTests/external/, measuring before that would complicate the script.)

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYc4%2BJviY7%2BVq4QtjrOVF3Sc5hYSVx_2acs%3DyxhKQsK86g%40mail.gmail.com.

[Philip Jägenstedt]

I told Domenic about this number today and he pointed out the same thing, that we have lots of upstream contributions as well. It's actually a very impressive number of non-merge wpt commits since Jan 17 from people I know are on Chromium in some way or another: 254

254/(1019-84+254) is 21%. We wanted to measure the update of the new export process, so we can't use that number, but it's still quite an impressive difference!

On Wed, Mar 22, 2017 at 10:59 PM Rick Byers <rby...@chromium.org> wrote:

Thanks for doing the analysis!  84 CLs modifying wpt since January is a great start (plus all the WPT work chromium folks do upstream in GitHub of course)!  This is better than I feared, though obviously still not as high as we'd like.  It'll be interesting to see how this changes as we start asking more about WPT in intents...

Rick

On Wed, Mar 22, 2017 at 5:01 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

I wrote a script to score our "20% of Blink CLs in LayoutTests/ are in LayoutTests/external/wpt/" KR and the result is that there were 1019 test changes, of which 84 were in wpt, just above 8%. We'd get a 0.41 score based on that, but let's measure again on March 31.

(Jan 17 is when LayoutTests/imported/ was moved to LayoutTests/external/, measuring before that would complicate the script.)

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYc4%2BJviY7%2BVq4QtjrOVF3Sc5hYSVx_2acs%3DyxhKQsK86g%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "platform-predictability" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAFUtAY-qxvb7q40PvOYhvQs1xBkxMAeQAybUPXoL5%2BetzTsACw%40mail.gmail.com.

[Dru Knox]

These numbers are awesome to see! Do we have any idea why there's such a large number of upstream changes not using the new tool? Is it just momentum that we need to move to the new system? Or are these changes things that wouldn't use the new import/export tool?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYfgM3CLU%3DsxastYt9h1S3qdPxxzr%2BnTVw6CVBYZkkZpWg%40mail.gmail.com.

[Philip Jägenstedt]

Certainly for most of the changes that Domenic is making, they're connected to spec changes instead of implementation changes. There I think it makes sense to continue using the upstream tooling, because the reviewers could be people working on any browser engine, or no browser engine.

Some of the changes are probably things that people could do in Chromium now, though.

We have to pick a new number for Q2. I think that everything is order so that people can just use wpt by default and that we've advertised it well enough, so I'd say 50%.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAJcfO47sNbJ5%3D%3Dynx-6xqEmXvEyPjr9Urr8gGpffp3P6Nqek7Q%40mail.gmail.com.

[Philip Jägenstedt]

An update for Q1 scoring, from Jan 17 to March 31 the numbers are 106/1209~=9%, so we'll score Q1 as 0.44.

[Domenic Denicola]

To me it really seems like this number is not measuring the right thing. I can understand wanting to stick with the existing method of measurement for Q1’s score, but for Q2 I’d suggest somehow taking into account all the work people are doing directly in the web platform tests repo.

 

A number of people are doing this as part of their Blink work, e.g. because they want wider review, instead of working in Blink only. That is, the workflow is to first commit a change to WPT on GitHub, then wait for the auto-importer to import the failing tests, then have their CL either fix the failures and remove the -expected.txt, or turn on importing of the new tests. Recent examples:

 

• All streams tests are developed upstream
• The StaticRange tests were developed upstream
• A lot of the IndexedDB work seems to be done upstream

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYfJHNib9G9jny1WzDWNw0yMviB-8u-nk4YpikMn%3DahYGA%40mail.gmail.com.

[Philip Jägenstedt]

The overall Q2 objective I had in mind is "WPT is suitable for testing all web-exposed features", so a different KR may be appropriate as well.

What would most directly demonstrate suitability is actual use, so one thing I've tried is to consider all changes that touch both Source/ and LayoutTests/, and see which proportion touches LayoutTests/external/wpt/. By also considering baseline changes, that would cover tests are written upstream and later made to pass. The number for March for that metric is ~15%.

I would like to measure how much we write web-platform-tests as well, but it's trickier. Should we count changes by Simon Pieters, Geoffrey Sneddon and Yoav Weiss? I think I did when I counted earlier in this thread, but it's not something one can write the script for up front. Maybe that's OK?