importing tests from the w3c, take three ...

[Dirk Pranke]

Hi all,

You are probably aware that we import a subset of the tests from the W3C's test repos, as per:

http://www.chromium.org/blink/importing-the-w3c-tests

and also aware that a few weeks ago I attempted to dramatically simplify that process but for various reasons my attempt failed.

I have done several rolls / imports of new tests since then, and concluded that my import process is too 

painful for anyone to want to do frequently, including me.

So, I propose that, instead, we move to a world where we simply *check the subsets of tests we care about directly into Blink*. 

To do so, one would run a slightly modified version of the update-w3c-deps script that currently exists, and post CLs for landing, just like any other change to a layout test. 

This means that any committer can update to new versions of the w3c tests that they care about at any time.

This is actually what WebKit is currently doing as well (as far as I know).

*Please speak up now if you think this is a bad idea, otherwise I plan to switch over in the next few days.*

A bit more color: 

In my original plan, we were running most if not all of the w3c tests, we were running them unmodified, updating the DEPS entries was easy, and so the import was mostly a no-op and would be done continuously. 

Since none of these things is currently true, and won't change in the near future unless someone else wants to work on the general "run more of the w3c tests directly" problem, I think it makes sense to change things to check in files directly as above.

[ If someone does want to work on this, feel free to follow up w/ me and I can outline what needs to be done ].

The two major downsides to this new approach over my original one are:

1) The Blink repo itself gets bigger. 

The checked-out text of the current set of w3c tests is about 2MB, as opposed to LayoutTests' total size of ~1GB, so this is not a big increase (the total git repo size of the two w3c repos, by comparison, is about 110 MB). We do not currently support importing pixel tests, so at least we don't have to worry about updating PNGs. 

2) We lose direct access to the history of the imported tests. 

However, it seems to me that anyone who likely actually cares about this is probably already directly working in the w3c repos and thus has access to the history elsewhere. 

Plus, since we rewrite the tests during the import, the history gets slightly mangled anyway, and since due to various technical glitches the merge commits of the rewritten files don't work right either, even the history gets mangled even more.

So, I don't think we're losing much by not having the history at this point.

There is an additional upside to the new approach: it makes supporting generic -expected.txt results for the imported tests possible without additional work (you can just check in and update baselines next to the test like any other layout test).

Note that when we eventually merge Blink and Chromium, much of the pain of the rolls will go away, and it might make sense to switch back to pulling w3c tests via DEPS.

Thanks,

-- Dirk

[Adam Klein]

As someone who's modified the tests upstream and had to wait for those changes to roll into Blink, I can appreciate that the existing way this was done was problematic. But I worry that failing to update "all the tests" regularly (not just on request) could reduce the benefit of running these tests. The case I'm imagining is:

- Browser vendor X (say, Mozilla) files a spec bug, gets the spec changed, updates their code, and submits a test to the test suite.

- No one on Blink notices.

Unfortunately I'm not familiar with all the issues that you ran into in take 2, so I'm willing to admit that it may be too hard (for whatever reason) to keep our tests relatively up-to-date with upstream. Just wanted to say that it seemed sad to not aim for that.

- Adam

[Dirk Pranke]

On Thu, Aug 28, 2014 at 5:04 PM, Adam Klein <ad...@chromium.org> wrote:

As someone who's modified the tests upstream and had to wait for those changes to roll into Blink, I can appreciate that the existing way this was done was problematic. But I worry that failing to update "all the tests" regularly (not just on request) could reduce the benefit of running these tests. The case I'm imagining is:

- Browser vendor X (say, Mozilla) files a spec bug, gets the spec changed, updates their code, and submits a test to the test suite.

- No one on Blink notices.

Unfortunately I'm not familiar with all the issues that you ran into in take 2, so I'm willing to admit that it may be too hard (for whatever reason) to keep our tests relatively up-to-date with upstream. Just wanted to say that it seemed sad to not aim for that.

Basically it turned out that gclient and bot_update didn't interact in a way that makes using a sub-DEPS file actually work the way we needed it to work. We could fix gclient and bot_update to work, but doing so is non-trivial and part of the reason why I didn't do things this way in the first place (and many the infra folks *really* don't want to add more complexity to gclient).

Note that it should be *much much easier* to automate updating the tests when we're checking things in directly than it currently is, because you will no longer have 3- or 5-sided patches. So, we don't have to abandon that aim.

(Basically, all it would take to automate things would be to stick update-w3c-deps into a cron job and post CLs any time we detected that there were new changes).

-- Dirk