Adding unit and tests to NSS and PSM

[Ludovic Hirlimann]

Hi,

I've been following NSS and PSM bugs over the last few years. Once in a
while an external contributors come up with a patch - and the review
process is slow mainly because the module owners fear that accepting the
patch might trigger regressions.
This is bad for the module and for the contributor. How can a
contributor stay motivated when a patch takes a very long time to get
into the tree or even to get reviewed.

Back when I started working on Thunderbird we had little to none
automated test. But the developers started adding more and more until we
reached a point where we thought we had enough test that we could
require new checkins to also include test. I've also noticed a decreased
in the time it took for reviews and the higher confidence from reviewers
on patches with tests. Having tests as make Thunderbird less prone to
regression and catching them a matter of hours.

I know, by reading various bug comments, that NSS as some machines
dedicated to testing somewhere. And from what I read It seems that these
tests suites are started manually.

I was wondering why there was no effort into bringing unit tests and
automated testing to the NSS and PSM couple ?

I think adding a automated test infrastructure to NSS and PSM would
make a nice project and would make the life of the current maintainers
a lot easier in the future.

>From what I know this is a not complete list of things that need to happen :

Decide on a unit test frame work for NSS
(<https://secure.wikimedia.org/wikipedia/en/wiki/List_of_unit_testing_frameworks#C>).

Integrate that unit test framework into NSS's build system and repport
failures in either buildbot/tinderbox/ current mozilla build tool (or
other tools used by other consumers of nss).
Start adding test - and from the size and complexity of the code that
seems like a lengthy task.
Implement a policy of adding tests for each new checkins
Make sure the test suite stays green

For PSM it might be bit more simple. As psm is more UI based , we could
reuse the worked done with mozmill and have tests the way Thunderbird as
them. This might mean build a special app that only display's PSM's UI
for testing.

Thoughts ? Did i miss something ?
How complicated do you think such project would be ?
Do you think it would be worth it ?

Ludo
--
Ludovic Hirlimann MozillaMessaging QA lead
https://wiki.mozilla.org/Thunderbird:Testing
http://www.spreadthunderbird.com/aff/79/2

[Nelson B Bolyard]

On 2011-08-03 02:22 PDT, Ludovic Hirlimann wrote:

> I know, by reading various bug comments, that NSS as some machines
> dedicated to testing somewhere. And from what I read It seems that these
> tests suites are started manually.

You have been misinformed.

For NSS, there is a collection of machines that continuously build and run
automated regression tests against it. See them at
http://tinderbox.mozilla.org/showbuilds.cgi?tree=NSS

The current set of regression tests is quite large and takes a LONG time
to run to completion. The top-most script may be seen at
http://mxr.mozilla.org/security/source/security/nss/tests/all.sh

Any person who builds NSS can run all.sh himself, if he has the patience.

Regardless, there are shortcomings in the test coverage and the addition
of more tests to the existing set would be welcome if it added coverage
and did not add too much to execution time.

I'm skeptical about test "frameworks". In my direct experience with an
NSS test development team in the past, test developers LOVE to develop
frameworks, but hate to develop tests that use those frameworks. So,
after literally man-years of framework development, we had a monster
framework and not a single test that ran in that framework. When the
framework was developed, and all that remained to be done was writing any
tests that run in it, the framework developers quit. IMO, we don't need
another framework. NSS needs real tests. Shell scripts make a fine
framework. NSS testing needs people who're willing to write code that
actually calls NSS API functions.

[Ian G]

On 3/08/11 8:38 PM, Nelson B Bolyard wrote:
> When the
> framework was developed, and all that remained to be done was writing any
> tests that run in it, the framework developers quit.

"You need a framework" is the 21st century version of that old epigram,
"I've got a bridge to sell you..." :)

> Shell scripts make a fine
> framework.

+1

iang

[Matt Wobensmith]

Hi Ludo -

I think this is very interesting. Thanks for starting the discussion.

I think the only reason it doesn't exist is that no one with time has stepped forward to claim it.

I've spoken with bsmith in the past about bringing more NSS testing in-house, but we haven't really had time to think about it. Clearly even just the NSS part is a major undertaking. If I weren't so busy, I'd consider taking it on.

I do agree, as you mention, that anything UI-based already has an established home in mozmill.

Best,

Matt Wobensmith
QA Engineer
Mozilla Corporation


Matt Wobensmith
QA Engineer
Mozilla Corporation

_______________________________________________
dev-quality mailing list
dev-q...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-quality