Feature Proposal: Async Testing

[Mikeal Rogers]

We made a conscious decision for the mozmill test framework to be
synchronous. This made perfect sense for functional tests.

Before 1.0 we added a unittest API that didn't raise exceptions so
that it could be run less synchronously, but this doesn't cover all
asynchronous cases. Now that the mozmill framework is being used more
and more to cover unittesting it sounds like a good time to create an
API for purely Asynchronous tests.

The primary limitation of the mozmill framework that makes
asynchronous testing difficult is the idea that only one test function
is running at any given time.
This isn't just a small implementation detail but a fundamental part
of the framework. It makes debugging worlds easier knowing that only
one test is running at any given time. This is something we can't
change without sacrificing more than we gain, but I think we can
change the definition of how a test finishes to allow for asynchronous
testing within the current framework limitations.

Right now, a test function is called, and when it returns the test is
done. Asynchronous tests are still running, or waiting for something,
at the time their test function returns. This leads me to consider an
API something like this.

testSomething = new MozMillAsyncTest("testSomething");

testSomething.tesOne = function ()
{setTimeout("testSomething.callback()", 15);};
testSomething.callback = function () {testSomething.finish();};

So we have this new object, that tells the mozmill framework that this
is a different kind of test. The framework knows to run this
differently and will run all functions that start with "test" that are
attached to testSomething. For the duration of the tests the "current"
mozmill test will be testSomething. The framework will then wait to
run another test until testSomething.finish() is called.

This is just the basic test layout. Inside of test functions like
testOne() I think that any exceptions should cause the framework to
fail all of testSomething and move on to the next test. You can use
all of the jum inside these functions so we don't need to define that
part of the API but there may be need for some other functionality to
be added to MozMillAsyncTest that I'm not immediately seeing.

Thoughts?

-Mikeal

[Clint Talbert]

This is how mochitest does asyncrhonous tests too, so I think it is a
good fit, and a good way to mix in the paradigms that people are already
familiar with. CCing dmose since his test cases spurred this discussion
to see if he has any thoughts on it.

-C