Thanks for the update, Dirk!
On Sat, Dec 21, 2013 at 8:31 AM, Dirk Pranke <
dpr...@chromium.org> wrote:
> We currently do not have a way to handle generic test baselines for the W3C
> tests, i.e., there are no -expected.txt files next to the tests themselves.
> (We can't put them next to the tests because that would require putting them
> in the w3c repos) You should be able to check in platform-specific
> expectations, but the baseline optimizer might get confused; I'll fix this
> soon.
>
> This is less of a problem than you might think, because I've modified
> run-webkit-tests to not require -expected.txt files at all for tests that
> use the "testharness.js" script for assertion-based tests. As long as all of
> the asserts pass, we'll consider the test as passing. We are also not
> importing any manual (aka pixel) tests, just text-only/script-based tests
> and reftests.
I suppose this means that we can't use a test that isn't fully
passing? That is a bit of a shame because when writing a test for
web-platforms-tests, you'd typically test against the spec and not
remove bits that happen to fail in Blink. As a concrete example, I've
been thinking about moving the GlobalEventHandlers tests to
web-platform-tests, but it sounds like I can't do that as long as
there are events in that interface which Blink doesn't support yet.
Correct?
> The tests that we do run are automatically modified during the import
> process (via the import-w3c-tests script) to prefix CSS attributes and
> update other things as needed to run inside content_shell. Ultimately I hope
> to modify enough things in content_shell directly so that we can just run
> the tests as-is and not need this step.
>
> Arguably, this whole process would be a lot easier if we just manually
> copied the tests into Blink. I'm not doing that because (a) I want to
> preserve the upstream repo history and (b) I think it's a bit more
> transparent and will ultimately allow us to more easily run new tests as
> they appear in the upstream repos. We'll see if this turns out to be a
> mistake :).
It might turn out to be a mistake, but I'm glad you did it this way.
Trying to figure out which repository is the most up to date between
Opera's internal tests, Blink's LayoutTests and web-platform-tests is
a waste of time, so one repo to rule them all is great!
> The process for uploading new tests to the W3C is entirely GitHub-centric
> and separate from running the tests. As we do more and more of this, I will
> add docs for this as well.
>
> I will upload much of this information onto
dev.chromium.org and send out a
> link to it.
>
> Let me know if you have any questions, concerns, or other feedback!
Where do you want us to be in a few years? Personally, I'd love it if
we could write tests in web-platform-tests directly and gradually move
more tests from LayoutTests to web-platform-tests. However, that would
either branching web-platform-tests, or making it very quick to get
tests into upstream and to then pull them back into Blink. (This
discussion is probably premature.)
Philip