Unit Testing For the Mozilla Toolkit
Benjamin Smedberg
meetings was improving the test automation and testability of our code. To
further this goal, I am instituting a new unit-test policy for all checkins
to the toolkit module.
The Importance of Testing
Unit testing offers many advantages:
* Testing gives code authors and reviewers confidence that patches produce
the correct results.
* Authoring testcases is a good impetus for developers to discover edge cases.
* Testing provides a good way to catch regressions quickly, and to make sure
that no regression will be repeated twice.
* Unit tests provide working examples for how to use an API and can
significantly aid documentation efforts.
Overall, integrated unit testing makes the cost and risk of any individual
change smaller. It will allow the project to make the major architectural
improvements of Mozilla 2 quickly and confidently.
Complete Coverage
In order for unit testing to achieve its promise, unit tests must be created
along with new code. The tests must be run regularly and the results must be
published where developers will notice and fix them. For Mozilla, this means
that most tests of the client code should be run during the make check pass
of the build system. The results of this pass are reported to the tinderbox
as TUnit.
I would like to have complete unit test coverage for all toolkit code. In
order to do this, I am instituting the additional checkin policies:
* All patches must include unit tests which are run during make check. Once
unit tests are committed, the in-testsuite+ flag should be set on the bug.
* Unit tests must be included in patches and reviewed just like any other code.
* If running the test during make check is not possible, consult with Dave
Liebrech or Robert Sayre to determine whether another test system is available.
* If a unit test framework is needed but is not yet available, the developer
should write appropriate test code and commit it. A bug should be filed on
the needed test framework. The in-testsuite? flag should be set on the bug
until the framework has been completed and the test code is running
automatically.
* Certain build-config or tooling bugs which don’t affect the actual product
cannot be usefully tested. These bugs should have the in-testsuite- flag set.
In addition, I will be soliciting volunteers to go through all the old bugs
in the toolkit; they will triage FIXED bugs with the in-testsuite? or
in-testsuite- flag, and then help produce appropriate testcases by 1 March,
2007. This is an ambitious goal, but I believe that we can achieve it, and
that it is important to get complete coverage before it becomes necessary to
branch for Mozilla 2.
Robert Sayre has graciously agreed to act as toolkit unit-test czar. If you
are a developer and you have questions about how to write a unit test or
what unit-testing frameworks are available, please first read the
documentation
<http://developer.mozilla.org/en/docs/Mozilla_automated_testing> and then
contact him (sayrer on IRC). Please don’t ask him to write the tests for you
;-) . Dave Liebrech (davel on IRC) is also available to answer question and
help with the design and implementation of unit-testing frameworks.
Review and Testing are Complementary
It is important to note that unit testing is not a substitute for review.
Testing is important because it ensures that the code does what it is
supposed to do. Review is important because it ensures that the code is
well-documented, understandable by a human being, and designed to work with
other code. Mozilla has traditionally been very good about requiring review;
I would like to build a similar culture around unit testing. Unit testing
and review are complementary activities and are both necessary for our
codebase to continue to improve.
Followup-To mozilla.dev.platform