extension based testing?
Axel Hecht
I'm in the process to write up a small extension that tests the firefox
help in terms of table-of-contents and search.
Basically, it's a spider that looks if there are any references in the
RDF files which are not in the xhtml. As the xhtml really only loads
without xml errors inside the browser, I created an extension.
Now, I guess there are more extensions that would like to do something
like this, for example davel's search tester.
I'd like to keep the work I have to do myself to a minimum and still
receive valuable input when I just load my entry page. On the other
hand, it'd be interesting to be able to run those automated.
So, here's my proposal:
- bootstrapping: no idea. I have no clue on how to tell 'the system'
that I wrote an extension and it should install it. I'm open on to
whether to install the extension in the profile or in the app. Both have
their cons and pros.
- running: the easiest way I came up with in terms of the runner
discovering tests in a somewhat flexible way is for the extension to
ship a default preference. That is both flexible and doesn't require
folks to write a service that can hook into the observer service or
whatever.
pref('mozilla.testing.l10n.help','chrome://....');
would be an example, and the runner would just enumerate everything
beneath 'mozilla.testing.'.
- reporting: via the console service. I'm not sure if I really like the
idea to have all communication go via Components.utils.reportError, may
be the simplest.
I'd require an extension to
Components.utils.reportError('TESTING:DONE');
or something similar, which would indicate to the runner that this test
is done, and it can head over an load the next test. The runner would
obviously register a listener on the console.
How does that sound? Anyone with cycles to actually pull that off?
Axel
Dave Liebreich
>
> I'd like to keep the work I have to do myself to a minimum and still
> receive valuable input when I just load my entry page. On the other
> hand, it'd be interesting to be able to run those automated.
> How does that sound? Anyone with cycles to actually pull that off?
I think it sounds really good. And I may have some cycles to get this
started.
Look for further posts from me in the next day or so, and feel free to
ping me if I haven't posted.
Thanks
-Dave
--
Dave Liebreich
Test Architect, Mozilla Corporation
Dave Liebreich
One issue concerning these types of tests is that I've tended to want to
run them against a large set of builds (40 locales * 3 platforms) at one
time. So I've worked on integrating the tests with something that can
easily be run by a freshly-installed app, and exits cleanly when the
test is complete.
Bob Clary has done a lot of work on this type of wrapper stuff, so he's
a good one to draw into this conversation.
Axel Hecht wrote:
> - bootstrapping: no idea. I have no clue on how to tell 'the system'
> that I wrote an extension and it should install it. I'm open on to
> whether to install the extension in the profile or in the app. Both have
> their cons and pros.
What are the cons and pros? I've been using -install-global-extension
since that's what Bob used.
>
> - running: the easiest way I came up with in terms of the runner
> discovering tests in a somewhat flexible way is for the extension to
> ship a default preference. That is both flexible and doesn't require
> folks to write a service that can hook into the observer service or
> whatever.
> pref('mozilla.testing.l10n.help','chrome://....');
> would be an example, and the runner would just enumerate everything
> beneath 'mozilla.testing.'.
I like this idea, since we could just drop a user.js into the profile
dir and start the app with a command-line flag like -runTests. David
Baron included the cli handler stuff in layout/tools/reftest, which
saves us the time of finding something to copy :-)
>
> - reporting: via the console service. I'm not sure if I really like the
> idea to have all communication go via Components.utils.reportError, may
> be the simplest.
> I'd require an extension to
> Components.utils.reportError('TESTING:DONE');
> or something similar, which would indicate to the runner that this test
> is done, and it can head over an load the next test. The runner would
> obviously register a listener on the console.
I like this idea for reporting results in some sort of xul-based
progress window. Since most of the tests I've been running have been
run against batches for builds, I've been using dump() and capturing
stdout/stderr in the wrapper.
>
> How does that sound? Anyone with cycles to actually pull that off?
>
> Axel
I'm off to PNSQC next week, and need to wrap up a bunch of other stuff
before I leave. If anyone makes progress on this stuff, please post a
pointer to it here in this thread.