Cross browser automated tests
Sylvain Pasche
Mozilla has developed a large set of automated tests. Apparently, a non
negligible part of these tests are not Mozilla specific and could be run
on other browser implementation.
This made me think about the following idea: having a repository of
browser independent automated tests. Each browser developer could
contribute tests to it and use these tests for testing their product.
Existing test suites that could be run in an automated way could be
imported into it (something that has already been done by Mozilla/WebKit
for the dom tests for instance). The technical details of the testing
environment and API would need to be discussed.
These tests could be run automatically on today's available browsers.
With the test results published, Web developers could check what feature
(if covered by tests) are implemented (and to what extent) in a given
browser.
What do you think about it? Would there be interest in such a project
from Mozilla?
Sylvain
Ray Kiddy
I see nobody has answered. I am outside MoCo but I have a feeling that
they want more automated tests for Firefox and thinking about which of
those tests may not be specific to Firefox is not very interesting.
You can always build, yourself, a database of tests that are available
from Mozilla and then add information to that database. It might even be
interesting to look at the other tests for other browsers. If you found
tests there that would be useful for Firefox, and MoCo was able to run
the tests, that might actually generate some interest.
Do you have information compiled about other automated testing efforts.
FYI, while I was contracting at Mozilla I compiled an inventory of
automated testing efforts. Part of that is available at:
http://wiki.mozilla.org/QA/TDAI/Inventory
cheers - ray
Jeff Walden
> Mozilla has developed a large set of automated tests. Apparently, a non
> negligible part of these tests are not Mozilla specific and could be run
> on other browser implementation.
Sure; on the other hand, it's very easy to accidentally rely on a Mozilla-specific feature when you write tests. Generators, array comprehensions, any of the array extras (modulo reimplementations like those on their MDC pages), Date.now, the list goes on and on. Avoiding these things takes real effort; I made this effort when writing postMessage tests recently, but it was more effort than you might expect to do so. (Someday I'd like to start a directory in Mochitest where "utility" scripts would live so you could get, say, a function which would only run a test if the browser is Mozilla-based, and start using that to generalize our tests, but that's a ways off yet in my schedule.)
Another problem concerns priorities. If you're going to punt some bug to later, you want to mark it as "this fails, return later". How do you distinguish these tests among the browsers that would want to run them, where one browser chooses to fix earlier than another? Doing it flexibly doesn't sound particularly easy.
> The technical details of the testing environment and API would need to
> be discussed.
Theoretically it all sounds fine, but the problem is agreeing upon some method of doing it that everyone finds acceptable. This is the biggest sticking point, as I see it.
> What do you think about it? Would there be interest in such a project
> from Mozilla?
I find the idea interesting, but I personally wouldn't invest too much effort in it. Nailing down details seems like a good way to spend a lot of time getting not much done, and I think I get more benefit from my time by spending it more or less as I do now. In the long run that might be different, but I think long run may just be *too* long, in this case. I'd throw in my two cents if someone else wanted to work on that, but I wouldn't put in a whole lot of work myself on it.
Jeff
Sylvain Pasche
Ray Kiddy wrote:
>
> I see nobody has answered. I am outside MoCo but I have a feeling that
> they want more automated tests for Firefox and thinking about which of
> those tests may not be specific to Firefox is not very interesting.
>
> You can always build, yourself, a database of tests that are available
> from Mozilla and then add information to that database. It might even be
> interesting to look at the other tests for other browsers. If you found
> tests there that would be useful for Firefox, and MoCo was able to run
> the tests, that might actually generate some interest.
>
> Do you have information compiled about other automated testing efforts.
For the moment, I just looked at Mozilla and WebKit automated tests.
I'll try to look at other automated efforts.
>
> FYI, while I was contracting at Mozilla I compiled an inventory of
> automated testing efforts. Part of that is available at:
>
> http://wiki.mozilla.org/QA/TDAI/Inventory
Thanks for the link, this is interesting.
Sylvain
Sylvain Pasche
Jeff Walden wrote:
> Sure; on the other hand, it's very easy to accidentally rely on a
> Mozilla-specific feature when you write tests. Generators, array
> comprehensions, any of the array extras (modulo reimplementations like
> those on their MDC pages), Date.now, the list goes on and on. Avoiding
> these things takes real effort; I made this effort when writing
> postMessage tests recently, but it was more effort than you might expect
> to do so. (Someday I'd like to start a directory in Mochitest where
> "utility" scripts would live so you could get, say, a function which
> would only run a test if the browser is Mozilla-based, and start using
> that to generalize our tests, but that's a ways off yet in my schedule.)
That's interesting to hear. On the other hand, there could be saved
effort by reusing tests written by other vendors or developers if that
works.
Tests could be tagged with the minimum Javascript version they require,
so they could run on other browsers once they've implemented that
version, but that wouldn't be a short term solution.
> Another problem concerns priorities. If you're going to punt some bug
> to later, you want to mark it as "this fails, return later". How do you
> distinguish these tests among the browsers that would want to run them,
> where one browser chooses to fix earlier than another? Doing it
> flexibly doesn't sound particularly easy.
If I've understood correctly, this is the role of the todo() that are in
Mozilla tests. There could be something like this: there are no todo()
in the test repository, but each vendor would manage a todo matrix
separately. This matrix could blacklists whole tests, or a specific
check in a test.
Now the problem is how to identify a specific check in a test. Say we
have the following checks:
is(evt.data, data, "unexpected data");
is(evt.origin, origin, "unexpected origin");
is(evt.lastEventId, lastEventId, "unexpected lastEventId");
Where we want to identify the second check as being a todo().
1) identify by the failure message: the trouble is that there could be
duplicates
2) identify by check sequence (second check here). This doesn't require
using an unique identifier for the check but is fragile and can be
painful to deal with in case of loops
3) Add an unique identifier in the check
is(evt.data, data, "check 1: unexpected data");
is(evt.origin, origin, "check 2: unexpected origin");
is(evt.lastEventId, lastEventId, "check 3: unexpected lastEventId");
This is nice for identifying checks, but requires more effort to add
these identifiers.
>
>> The technical details of the testing environment and API would need to
>> be discussed.
>
> Theoretically it all sounds fine, but the problem is agreeing upon some
> method of doing it that everyone finds acceptable. This is the biggest
> sticking point, as I see it.
>
Yes, that one can be difficult. One idea would be to make the
requirements in term of API as unobtrusive as possible:
* One .js file to include
* A set of functions that can be called for performing the checks and
life cycle functions (end of tests, ...)
> I find the idea interesting, but I personally wouldn't invest too much
> effort in it. Nailing down details seems like a good way to spend a lot
> of time getting not much done, and I think I get more benefit from my
> time by spending it more or less as I do now. In the long run that
> might be different, but I think long run may just be *too* long, in this
> case. I'd throw in my two cents if someone else wanted to work on that,
> but I wouldn't put in a whole lot of work myself on it.
I'm interested to try to move things forward. I've already done some
experiments by running mochitests and reftests on other browsers.
Results are visible on http://www.browsertests.org/. Lots of results are
not correct because I didn't run them through runtests.pl, and I didn't
implement .sjs, ^headers^ or a proxy for cross domain testing.
I tried to extract some webkit tests that could potentially be cross
browser (tests 1-4307).
Sylvain
Jeff Walden
> If I've understood correctly, this is the role of the todo() that are in
> Mozilla tests.
Bingo.
> Now the problem is how to identify a specific check in a test. Say we
> have the following checks:
>
> is(evt.data, data, "unexpected data");
> is(evt.origin, origin, "unexpected origin");
> is(evt.lastEventId, lastEventId, "unexpected lastEventId");
>
> Where we want to identify the second check as being a todo().
> 1) identify by the failure message: the trouble is that there could be
> duplicates
> 2) identify by check sequence (second check here). This doesn't require
> using an unique identifier for the check but is fragile and can be
> painful to deal with in case of loops
> 3) Add an unique identifier in the check
> is(evt.data, data, "check 1: unexpected data");
> is(evt.origin, origin, "check 2: unexpected origin");
> is(evt.lastEventId, lastEventId, "check 3: unexpected lastEventId");
> This is nice for identifying checks, but requires more effort to add
> these identifiers.
Yeah, loops were the big issue I noticed. The postMessage tests where I sent in various data to test targetOrigin-handling are a pathological case of this, where all the information about what should happen is in the array of tests far away from the actual code that executes. Hard-coding the list of checks as in #2 is what WebKit does, but it essentially requires generating the test results from the implementation, which I'd rather not do. (What if a test is accidentally skipped but you didn't notice in the middle of 200 others before you committed?)
Also, you're going to be try/catching this stuff quite liberally if you want any browser to be able to run it without prematurely failing and causing a test hang.
> I'm interested to try to move things forward. I've already done some
> experiments by running mochitests and reftests on other browsers.
Nice; this is something I can see being in WHATWG's purview for helping everyone be able to use a centralized test repository. I just think I'll be more productive in a global sense doing what I do well and leaving this to others who either can do it better than me or who can't advance Mozilla's testing situation as well as I can. I'm basically applying the comparative advantage concept to my time allocation to the two testing methods.
> Results are visible on http://www.browsertests.org/. Lots of results are
> not correct because I didn't run them through runtests.pl, and I didn't
> implement .sjs, ^headers^ or a proxy for cross domain testing.
If you can run Mochitests in Mozilla, running them in another browser is easy (or at least it is now; some SSL/HTTPS/certificate testing coming down the pipeline could easily make this harder, since we'll be auto-importing a set of potentially runtime-generated certs for the tests). Just open about:config search for proxy, and find the proxy autoconfig result. Copy that value, stripping the data:mimetype, part, into a file and point the target browser at it as a PAC URL, then open http://localhost:8888/tests/ and go to town. Making the tests exportable as I did manually with the postMessage tests is a touch harder (gotta generate an index.html) and may require effort to avoid Mozilla-specific tests (test_postMessage_jar.html for one), but this gets you running them at least.
The web server part of this is the big/hard part to replicate in the harness; however, with fairly little effort you could create a xulrunner app that included the server and did all the right stuff. The big problem is its current limitations, like not being able to read the body of requests or do asynchronous responses, which I'll address eventually, but it will be awhile -- at this point possibly not even this year. Also there's the necessity of knowing Mozilla APIs if you want to do anything fancy in the request handlers. I'm really not sure what's the right answer for any test requiring an interactive server response, for something that's reasonably pluggable.
> I tried to extract some webkit tests that could potentially be cross
> browser (tests 1-4307).
Interesting. I've considered eventually looking into what they have but have never found the time.
Jeff