Testing of OOP (TOOP) FAQ, December
Doug Shaker
Testing of Object-Oriented Programming (TOOP) FAQ/Resource Summary
Posted to comp.object, comp.lang.c++, comp.lang.smalltalk and
comp.software.testing.
Last revised on 93.11.29. The most notable change is that a couple of
articles were added to the Written Material section (the Overbeck
and Dorman articles from EuroSTAR '93).
> What?
______________
This is a summary of resources on the Testing of Object-Oriented
Programming that have been mentioned to me over the net, in email,
or other means. Sections include Written Material, Courses, and
Software. It is kind of like an FAQ, though it isn't organized
that way.
> Who?
______________
I work for a Unix software house, Qualix Group, in the US. Here is
my sig:
- Doug Shaker
voice: 415/572-0200
fax: 415/572-1300
email: dsh...@qualix.com
mail: Qualix Group
1900 S. Norfolk St., #224
San Mateo, CA 94403
I am NOT a researcher on the testing of object-oriented programming.
I just collate the stuff that is sent to me by people who REALLY know
something. See the section "ACKs" at the end.
I just think it is important.
> Why?
_______________
Why is this important? If classes are really to be reused in
confidence, they must be blatantly correct. The classes must be easily
testable during initial evaluation by the client programmer. They must
also be testable under different OS configurations, different compiler
optimizations, etc. This means that testing modules must be
constructed in a way which is recognized as correct and the modules
must be shipped with the class libraries.
As soon as one major class library vendor starts to ship real test code
with their libraries, all of the other vendors will be forced, by
market pressure, to do so as well, or face market share erosion. Think
about it. If you had to recommend a class library to a committee that
was choosing a basis for the next five years of work, wouldn't you feel
safer with a class library that could be auto-tested in your
environment?
> Written Material
_____________________
Berard, Edward. Essays on Object-Oriented Software Engineering.
Prentice-Hall, Englewood Cliffs, NJ. $35.
This book has two chapters on testing of object-oriented software,
focusing on how to do it.
Berard, Edward. Project Management Handbook. Must be purchased
direct from Berard Software Engineering, Ltd., 902 Wind River
Lane, Suite 203, Gaithersburg, Maryland 20878. $225.
The book focuses on the management of OOP projects. It
includes one chapter on testing OO software and one chapter
on quality assurance.
Bezier, Boris, "Software Testing Techniques", 2nd edition, Van Nostrand
Reinhold, 1990, 503pp, $43, ISBN 0-442-20672-0. While this is
not specifically about testing of OOP, it is mentioned so often
by so many people as a definitive software testing work, that
I have to mention it anyway.
Cheatham Thomas J., and Lee Mellinger, "Testing Object-Oriented
Software Systems", Proceedings of the 18th ACM Annual Computer
Science Conference, ACM, Inc., New York, NY, 1990, pp. 161-165.
Doong, Roong-Ko and Phyllis G. Frankl, "Case Studies on Testing
Object-Oriented Programs", Proceedings of the 4th Symposium on
Testing, Analysis, and Verification (TAV4), 1991, ACM, Inc.,
New York, NY, 1991, pp. 165-177.
Dorman, Misha. Unit Testing of C++ Objects. In: Proceedings of the 1st
European International Conference on Software Testing Analysis &
Review (EuroSTAR'93) 1993, London, UK 25-28 October 1993. Pp. 71-
101. Copies of the paper are available from the author at IPL,
Eveleigh House, Grove Street, Bath BA1 5LR, United Kingdom
(email: mis...@iplbath.demon.co.uk).
Fiedler, Steven P., "Object-Oriented Unit Testing", Hewlett-Packard
Journal, April, 1989, pp. 69-74.
Firesmith, D.G., "Testing Object-Oriented Software", Proceedings
of 11th. TOOLS USA Conference, Santa Barbara, Aug 1993, pp 407-426.
Frankl, Phyllis G. and Roong-Ko Doong, "Tools for Testing
Object-Oriented Programs", Proceedings of the 8th Pacific
Northwest Conference on Software Quality, 1990, pp. 309-324.
One author can be reached at either of the following
email addresses: pfr...@poly.edu or phy...@morph.poly.edu.
Graham, J.A., Drakeford, A.C.T., Turner, C.D. 1993. The Verification,
Validation and Testing of Object Oriented Systems, BT Technol
J. Vol 11, No 3. One author's email address is
jgr...@axion.bt.co.uk.
Harrold, Mary Jean, John D. McGregor, and Kevin J. Fitzpatrick,
"Incremental Testing of Object-Oriented Class Structures",
International Conference on Software Engineering, May, 1992,
ACM, Inc., pp. 68 - 80.
Hoffman, Daniel and Paul Strooper. A Case Study in Class Testing.
To be Presented at the IBM Center for Advanced Studies Fall
Conference, October 1993, Toronto. Email addresses for authors
are dhof...@csr.uvic.ca and pst...@cs.uq.oz.au. Describes an
approach to testing which the authors call Testgraphs. An
example is worked out in C++ which tests a commercial class.
Hoffman, D. M. A CASE Study in Module Testing. In Proc. Conf. Software
Maintenance, pp. 100-105. IEEE Computer Society, October 1989.
Hoffman, D.M. and P.A. Strooper. Graph-Based Class Testing. In
7th Australian Software Engineering Conference (to appear), 1993.
Klimas, Edward "Quality Assurance Issues for Smalltalk Based Applications",
The Smalltalk Report, Vol. 1, No. 9, pp.3-7. The author's
email address is "ac...@cleveland.freenet.edu".
Lakos, John S. "Designing-In Quality in Large C++ Projects" Presented
at the 10th Annual Pacific Northwest Software Quality Conference,
Portland, Oregon, October 21, 1993. Abstract:
The focus of this paper is on ensuring quality by
designing software that avoids acyclic component
dependencies. This in-turn permits incremental,
hierarchical testing. The importance of good physical
design becomes a key factor only for large and very
large projects. Intuition gained from smaller projects
leads to errors in large designs. Compile-coupling
("Insulation") is also discussed.
Copies of the postscript file can be obtained by sending email
to "john_...@warren.mentorg.com".
Leavens, G. T., "Modular Specification and Verification of
Object-Oriented Programs", IEEE Software, July 1991, pp. 72-80.
Love, Tom. Object Lessons. SIGS Books, 588 Broadway #604, New York, NY
10012. $49.
This book eloquently elucidates the need for testing of object-
oriented code and has a chapter on how it was done at Stepstone
during the first release of their initial class library.
McGregor & Sykes. Object-Oriented Software Development, Van Nostrand
Reinhold, 1992. This has a chapter on testing of OO programs.
Marick, Brian. The Craft of Software Testing, Prentice-Hall, in press.
Makes the argument that testing of object-oriented software can
be treated as a special case of testing of software which retains
state and which is reused. The author can be reached at
in...@testing.com.
Marick, Brian. "Testing Software that Reuses", Technical Note 2, Testing
Foundations, Champaign, Illinois, 1992. Copies may be obtainable
via email. The author can be reached at in...@testing.com.
Murphy, G.C., Wong, P. 1992, Towards a Testing Methodology for
Object Oriented Systems, M.P.R Teltech Ltd. A poster at the
Conference on Object Oriented Programming Systems, Languages
and Applications ACM. Copies of this paper can be obtained
through town...@mprgate.mpr.ca.
Murphy, G. and P. Wong. Object-Oriented Systems Testing Methodology: An
Overview. Technical Report TR92-0656, MPR Teltech Ltd., October
1992.
Overbeck, Jan. Testing Object-Oriented Software - State of the Art and
Research Directions. In: Proceedings of the First European
International Conference on Software Testing, Analysis &
Review 1993 (EuroSTAR'93), London UK, October 25-28, 1993.
Pp. 245-270. The author's email address is
"ov...@vexpert.dbai.tuwein.ac.at" and the paper can be ftp'd from
mira.dbai.tuwien.ac.at as "/pub/overbeck/survey.ps". The
complete proceedings can also be purchased from EuroSTAR
Conference, Ltd., P.O. Box 2645, West Kensington, London, W14
9HD, United Kingdom, Fax: +44-71-385-1596.
Perry, D.E. and G.E. Kaiser, "Adequate Testing and Object-Oriented
Programming", Journal of Object-Oriented Programming,
2(5):13-19, Jan/Feb 1990.
Purchase, Jan A. and Russel L. Winder, "Debugging tools for
object-oriented programming", Journal of Object-Oriented
Programming, June, 1991, Vol. 4, No. 3, pp. 10 - 27.
Smith, M. D. and D. J. Robson, " A Framework for Testing Object-Oriented
Programs", JOOP, 5(3):45-53, June 1992.
Describes ways in which the usual approach to software testing
could be adapted for object-oriented software.
This paper, or one with the same title and authors, is
available by anonymouns ftp from vega.dur.ac.uk as
"/pub/papers/foot.dvi".
Smith, M. D. and D. J. Robson, "Object-Oriented Programming - the
Problems of Validation", Proceedings of the 6th International
Conference on Software Maintenance 1990, IEEE Computer Society
Press, Los Alamitos, CA., pp. 272-281.
Taylor, David. "A quality-first program for object technology", Object
Magazine, Vol. 2, No. 2, July-August 1992, pp17-18. SIGs
Publications. The article talks some about why testing is
important for OOP and describes one quality program.
Theilen, David. "No Bugs. Delivering error free code in C and C++.",
Addison-Wesley, 1992, ISBN:0-201-60890-1.
Turner, C. D. and D. J. Robson, "The Testing of Object-Oriented Programs",
Technical Report TR-13/92, Computer Science Division, School of
Engineering and Computer Sciences (SECS), University of Durham,
England.
Includes a survey of existing literature on testing of OO
programs. Testing of OOP is compared with traditional software
testing. A state-based approach is described.
This paper is available by anonymous ftp from vega.dur.ac.uk in
/pub/papers. Get "toop.ps.Z" for A4 paper and "toopus.ps.Z" for
US letter paper formatting.
Turner, C. D. and D. J. Robson, "A Suite of Tools for the State-Based
Testing of Object-Oriented Programs", Technical Report
TR-14/92, Computer Science Division, School of Engineering and
Computer Science (SECS), University of Durham, Durham,
England. Describes a series of tools for the generation and
execution of test cases for OOP. These tools assume a
state-based testing approach.
This paper is available by anonymous ftp from vega.dur.ac.uk in
/pub/papers. Get "tools.ps.Z" for A4 paper formatting or get
"toolsus.ps.Z" for US letter formatting.
Turner, C. D. and D. J. Robson, "Guidance for the Testing of Object-
Oriented Programs", Technical Report TR-2/93, Computer Science
Division, School of Engineering and Computer Science (SECS),
University of Durham, Durham, England. Discusses different
methods of making class declarations and the implications of
those methods for testing.
This paper is available by anonymous ftp from vega.dur.ac.uk in
/pub/papers. Get "guide.ps.Z" for A4 paper formatting or get
"guideus.ps.Z" for US letter formatting.
Turner, C. D. and D. J. Robson, "State-Based Testing and Inheritance",
Technical Report TR-1/93, Computer Science Division, School of
Engineering and Computer Science (SECS), University of Durham,
Durham, England.
Discusses the implications of inheritance for testing,
particularily incremental testing.
This paper is available by anonymous ftp from vega.dur.ac.uk in
/pub/papers. Get toopinht.ps.Z" for A4 paper formatting or get
"toopinhtus.ps.Z" for US letter formatting.
Wong, P. Automated Class Exerciser (ACE) User's Guide. Technical
Report TR92-0655, MPR Teltech Ltd., September 1992.
> Courses
__________________________
Berard Software Engineering, Inc. teaches a seminar on Testing of
Object-Oriented Software (TOOS). Call 301-417-9884 for details.
Quality Fractals, Inc. has a course called "Testing Object-Oriented
Software". Contact: 508-359-7273 (Box 337, Medfield, MA 02052). The
course is taught by Shel Siegel of YESS!, Inc. Contact: 916-944-1032.
> Software
__________________________
There is a smalltalk class library in the Univ. of Illinois archives
which includes a simple Tester class written by Bruce Samuelson
(br...@utafll.uta.edu). It is a general superclass for application
specific classes that test non-interactive objects such as trees,
collections, or numbers. It is not suitable for testing user interface
components such as windows, cursors, or scroll bars. The filein
includes Tree classes, Tester itself, and subclasses of Tester that are
used to validate the Tree classes. For ParcPlace Smalltalk (ObjectWorks
4.1 and VisualWorks 1.0). To get it ftp the file
"/pub/st80_vw/TreeLW1.1" from st.cs.uiuc.edu.
IPL Ltd. (in the UK) has a testing tool called Cantata which allows for
testing C++, but as far as I am able to determine, it has no special
features for C++ testing. From the product literature:
Cantata allows testing to be performed in an intuitive way
making the tool exceptionally easy to use and productive in
operation. Cantata is suitable for testing software written in
either C or C++.
Cantata provides comprehensive facilities for all forms of
dynamic testing, including: functional testing, structural
testing, unit testing and integration testing. Cantata has been
specifically designed to operate in both host and target
systems and so allow full portability of tests between these
environments.
For more information contact IPL:
IPL Ltd.
Eveleigh House, Grove Street,
Bath BA1 5LR
UK
(0225) 444888
(0225) 444400 (FAX)
email: sh...@iplbath.demon.co.uk
TestCenter from CenterLine will do coverage testing of C++ (and C)
code. Also does some memory debugging (similar to Purify) and regression
testing. Highlights from CenterLine literature:
*Automatic run-time error-checking on executables to enhance quality
*Automatic memory leak detection on executables to optimize memory use
*Graphical test coverage to highlight any code not executed during test runs
*Intuitive GUI for easy test analysis
*Programmatic interface to output files and cumulative code coverage
to support batch-mode and regression testing
*No recompilation needed, resulting in quick turnaround
*Complete C and C++ language support
*Integration with leading programming tools for maximum productivity gains
MicroTech Pacific Research (mpr.ca) has a C++ class testing tool called
ACE (Automated Class Exerciser) which is available under non-disclosure
agreement. It is not currently for sale. If you are interested,
contact Paul Townsend, town...@mprgate.mpr.ca.
Software Research Inc. (625 Third St, San Francisco, CA 94107-1997,
voice: 1-415-957-1441, email: in...@soft.com) has a coverage tool for C++
that is called tcat++. It is an extension of SRI's tcat program.
Quality Assured Software Engineering (938 Willowleaf Dr., Suite 2806,
San Jose, CA 95128, voice: 1-408-298-3824 ) has a coverage tool for
C and C++ called MetaC. It also dones some syntax checking and memory
allocation checking.
A group of volunteers is building a C++ test harness for the automated
testing of C++, C and Perl programs. The system is called ETET (Extended
Test Environment Toolkit). To join the group of volunteers, send email to
etet_s...@uel.co.uk
The software is available via anonymous FTP from bright.ecs.soton.ac.uk
(152.78.64.201) as "/pub/etet/etet1.10.1.tar.Z". They are looking for
other FTP sites - send email to the above address if you can provide
one. This is a beta release and _should_ compile on any POSIX.1 system.
As much of this work is being done by SunSoft, my guess is that the
software will have the fewest problems on SunOS or Solaris releases.
> ACKs
___________________
Thanks to the following for helping assemble this list:
Benjamin C. Cohen, bco...@scdt.intel.com
Brian Marick, mar...@hal.cs.uiuc.edu
Bruce Samuleson, br...@utafll.uta.edu
Daniel M. Hoffman, dhof...@uvunix.uvic.ca
Edward Klimas, ac...@cleveland.freenet.edu
John Graham, J.Gr...@axion.bt.co.uk
Jim Youlio, j...@bse.com
Jeffery Brown, jeffre...@medtronic.com
Lars Jonsson, kon...@etna.ericsson.se
Manfred Scheifert, ch_s...@rcvie.co.at
Mark Swanson, mswa...@mechmail.cv.com
Mary L. Schweizer, ma...@gdwest.gd.com
Michael Einkauf, Michael...@iegate.mitre.org
Paul Townsend, town...@mprgate.mpr.ca
Phyllis G. Frankl, pfr...@poly.edu or phy...@morph.poly.edu
Rachel Harrison, r...@ecs.soton.ac.uk
Risto Hakli, r...@tko.vtt.fi
Russ Hopler, ru...@bse.com
Stephane Barbey, bar...@di.epfl.ch
Tony Reis, to...@hpsadln.sr.hp.com
Yawar Ali, ya...@bnr.ca
Special thanks to Risto Hakli, who has contributed many of the references.