comp.software.testing Frequently Asked Questions (FAQ)
Raymond Rivest
Posting-Frequency: monthly
Last-modified: 2002/08/06
Version: $Revision: 3.9 $
Maintainer: cst...@crim.ca
URL: http://www.faqs.org/faqs/software-eng/testing-faq/
http://www.crim.ca/ctl/cst.FAQ.html
http://www.cigital.com/c.s.t.faq.html
comp.software.testing FAQ
and information resource
This is the Frequently Asked Questions list for comp.software.testing.
This FAQ is cross-posted to comp.software-eng. Please don't hesitate
to send corrections, additions, and other comments to
cst...@crim.ca.
Be sure to use the tag of FAQ: in the subject line, for example,
"Subject: FAQ: Modification request...."
The latest version of this FAQ may be retrieved using Usenet, the World
Wide Web, ftp, email, gopher, or even telnet. See the FAQ "Introduction
to the *.answers newsgroups" in news.announce.newusers or
<ftp://rtfm.mit.edu/pub/usenet/news.answers/news-answers/introduction>
for details. Please check the date above - if this file is more than a
month old, it is obsolete. The main ftp site for the FAQ is
<ftp://rtfm.mit.edu/pub/usenet/news.answers/software-eng/testing-faq>.
The folks at CRIM Software Test Center sponsor a web site
for the FAQ at http://www.crim.ca/ctl/cst.FAQ.html.
The folks at Cigital sponsor a web site for the FAQ, at
<http://www.cigital.com/c.s.t.faq.html>.
This file is Copyright 2001, 2002 by Raymond Rivest. Danny R. Faught is the
original author. Permission is granted to distribute this file unedited
and in its entirety, provided the "Date" header is no more than one month
earlier than the date of distribution. This information is provided
without any express or implied warranty.
------------------------------
Subject: 1. Table of Contents
1. Table of Contents
2. Change Log
3. What is this newsgroup about?
4. Netiquette
5. Beta testing
6. What other FAQs are relevant?
7. What other resources are available?
8. World Wide Web resources
9. What's the difference between QA and testing?
10. I'm looking for a test tool...
11. How do I find information about testing object-oriented programs?
12. How do I test web-related applications?
13. What is black box/white box testing?
14. What are unit, component and integration testing?
15. What's the difference between load and stress testing ?
16. Should we discuss bug tracking tools in this newsgroup?
17. What kind of salary should a tester make?
18. Where can I find sample test plans?
19. What is the best tester to developer ratio?
20. Conferences
21. Periodicals
22. Books
23. Bibliographic resources
24. Organizations
25. Contributors
------------------------------
Subject: 2. Change Log
This section will highlight notable additions, deletions, and changes to
the FAQ.
Changed in version 3.:
o Some minor cosmetic changes.
o Added CRIM STC in section 24.
Changed in version 3.8:
o Added Extreme Programming mini-FAQ link in section 6.
Changed in version 3.7:
o Finally cleaned up Danny's e-mail to tejasconsulting :)
o Removed "The Outlook" is no longer available from McCabe? (subject 22).
o Added a commercial newsletter to subject 22: The Payne Report,
http://www.cigital.com/paynereport/.
o Changed the comp.software.config-mgmt url to
http://www.daveeaton.com/scm/PMTools.html.
o Added http://www.QALinks.com and http://www.TestingFAQs.org
in subject 8.
o Added Die Software Testing Website http://www.softwaretesting.de
in suject 8.
o Added LogiGear web page at http://www.qacity.com/ in subject 8.
o Added Tejas Software Consulting Newsletter in subject 21.
o Added a new subject tag in section 4 (netiquette) for job searcher (CV:).
o Added a link to http://www.xprogramming.com/software.htm
in subject 7 about eXtreme programming and testing.
o Added two references about security testing in subject 7.
o Added Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid
in subject 12.
o Updated a broken link for http://www.asq.org/pub/ in subject 21
o Added a quick reference defining what is a "smoke test" in subject 7.
Changed in version 3.6:
o Changed reference from www.dejanews.com to groups.google.com.
Google acquired dejanews !
Changed in version 3.5:
o Inserted a new Section 15 and moved the rest downward.
What's the difference between load and stress testing ?
Changed in version 3.4:
o Reordered the Table of Content. Section 7 to 4, 10 to 5 and moved the
rest downward to explain the purpose of comp.software.testing from
the start.
o Changed references of Danny Faught's FAQs from his home page to
http://www.testingfaqs.org/ in Section 10 (a forgotten link).
o Added TESTCOM conference to Section 19.
o Added reference to the defect tracking tool FAQ in Section 15.
o Added StickyMinds.com in Section 8.
o Added a book reference in Section 12.
o Added the StickyMinds.com template section in Section 17
o Added a reference to comp.software.measurement newsgroup in
Section 7.
Changed in version 3.3:
o Added the mention of subject tag for cst-faq alias because of spams.
o Changed references of Danny Faught's FAQs from his home page to
http://www.testingfaqs.org/
o Added reference to BetaSoft's QAForums in section 5.
Changed in version 3.2:
o Changed references of Danny Faught's FAQs to his home page.
o Changed Danny Faught's email.
o Added LatinStar 2001 conference.
o Changed STQE booklist to stickyminds.
o Added a new URL for the html FAQ.
Changed in version 3.1:
o Changed references of cst...@cigital.com to cst...@crim.ca.
Changed in version 3.0:
o Head FAQ maintainer responsibilities transferred from Danny Faught to
Raymond Rivest. Have fun with it, Raymond!
Changed in version 2.7:
o Delegated some items to Cigital's SRM Hotlist, so I don't have to
maintain them in two different places. This includes the list of mailing
lists (subject 5), web pointers and dates for conferences (subject 19),
and the entire book list (subject 21). This will also help to make the
FAQ somewhat less bulky.
o Removed the Triangle Quality Council, which is being phased out.
Also removed Association for Software Testing Australia, which no
longer seems to exist. (subject 23)
Changed in version 2.6:
o Reliable Software Technologies is now Cigital, Inc. Changed all
RST references to Cigital.
o Reflected the change from American Programmer to Cutter IT Journal.
(subject 20)
o Removed information about the cst-improve list, which is no longer
in use. (subject 5)
o Updated contact information for Software Practitioner. (subject 20)
------------------------------
Subject: 3. What is this newsgroup about?
If you're new to Usenet, please read through the FAQs in
news.announce.newusers and hang out in news.newusers.questions for a
while before you consider posting.
If you do not know how to subscribe to comp.software.testing, you
first need to get Usenet access from an Internet service provider or
your company, and you need newsreader software. Different service
providers support different newsreaders, so there is no general
answer. Ask your service provider or local system administrator to
help. If all else fails, use a web browser to access Usenet at
<http://groups.google.com/groups
?hl=en&lr=&safe=off&group=comp.software.testing>.
The original charter for comp.software.testing can be found at
<http://tsunami.jpl.nasa.gov/TEL/docs/cst-charter.html>. Here's an
excerpt (with corrections):
CONTENTS:
For the purpose of this discussion, we will assume that a system is
composed of hardware, software, people, and procedures. The
proposed group should be chartered to include discussions
characterized by the following list of subjects, compiled from
preliminary discussions:
system test automation
system test planning
system test optimization (e.g. minimize regression testing)
test tools
test metrics
measurement technology
nature of testing under various development models
(e.g. object-oriented, real time, etc...)
testing in a rapid prototyping environment (i.e. sans spec)
relationship of various classes/types of tests to requirements, ...
conference and symposium announcements
the definition of "software testing" and its relationship to
SQA and debugging.
requirements traceability
risk-based testing
the most useful form of specifications and functional
requirements from the tester's point of view.
testing techniques, e.g., structured testing using control
flowgraphs and basis path testing, equivalence class
partitioning, boundary value analysis, cause-effect graphing,
path predicate testing, data flow testing, program slices,
data object state and usage analysis, data flow anomaly
analysis, and sensitivity analysis.
test coverage criteria, e.g., statement (C0), path, branch (C1),
module (S0), and call-pair (S1) coverage criteria.
All levels of testing for both hardware and software should be
considered appropriate subject matter. While it is likely that the
group will take on a software flavor early on, we should try to
generate interest and input from people with hardware and system
testing experience and perspective. It is especially important that
we foster discussion of *system* level testing issues, since this
is the weakest area of our collective knowledge.
------------------------------
Subject: 4. Netiquette
Did I mention that you should read the FAQs in news.announce.newusers if
you haven't done so lately? Also, you can reduce the chances of
embarrassing yourself by reading comp.software.testing for a while
before posting. At the very least, scan all the articles in the group
currently on your server to make sure your question hasn't already been
asked and answered.
Please help the newsgroup's participants find articles in the categories
they're interested in. If your posting falls into the categories below,
please put the three-letter tag at the beginning of your subject line.
This will also increase the likelihood that those wanting to read your
post will find it. For example, your subject line might read -
"Subject: ADV: Widget Tester 6.0 released".
JOB: Help wanted ads
ANN: Announcements for new books, publications
CFP: Call for papers or participants
CNF: Conference announcements
ADV: Other advertisements and commercial product announcements
CV: Curriculum Vitae, looking for a job
Job postings that are cross-posted to the various job-related
newsgroups should not be posted to comp.software.testing, since the
subject conventions for those groups do not allow them to be easily
identified as job postings, and the misc.jobs FAQ
(<ftp://rtfm.mit.edu/pub/usenet/news.answers/jobs/welcome-faq>)
specifies that job postings should not be cross-posted between the
job-related newsgroups and non job-related newsgroups.
Some recruiters refuse to follow the job posting policy. If you don't
want to read job postings, consider using a kill file to
systematically kill articles from these recruiters.
Many questions about a particular test tool only interest those readers
who use that tool. Please make sure to put the name of the test tool
in the subject line so readers can select the articles they read by
looking at the subject. In general, please use a descriptive subject
line.
If you post an article and later decide you shouldn't have posted it,
please cancel it. It is much better to correct the problem than to
send *another* post to apologize for the first one. Situations where
this would be appropriate include posting empty articles, multiple
copies of the same article, and any goof where you'd like to correct
something you posted shortly after it was sent. Look in your
newsreader documentation for help with cancelling an article; some also
have a "supercede" feature which makes it easy to correct a posting.
While the cancel feature has been common for quite some time, some
newer newsreaders unfortunately do not offer it. If this is the case,
complain to the author, and consider using a more reasonable
newsreader. Here's a manual method to cancel an article - start to
follow up to the errant posting. Change the subject to
"cmsg cancel <message-id>", where message-id is taken from the
References: header or the attribution. The body of the message is not
important. Post it, and the news system should interpret it as a
cancel request rather than a real posting. You can only do this for
an article you posted yourself.
People have also expressed concern about postings that are totally
off-topic. These postings are typically "spam" postings that go to most
of the thousands of newsgroups that exist. There is very little that an
unmoderated group can do proactively to prevent them. See
news.admin.net-abuse.usenet and
<http://www.tezcat.com/~gbyshenk/ive.been.spammed.html> for more
information.
Please don't post test messages to comp.software.testing. This group
is for discussing software testing, not for testing your news
software. If you want several automatic confirmations that your post
worked, post a test message to misc.test. There are other test groups
at various levels of the news hierarchy that all end in ".test". It's
best to try a local one first.
------------------------------
Subject: 5. Beta testing
Discussions about managing a beta test program are within the scope of
this newsgroup. However, companies looking for beta testers, and
aspiring beta testers looking for something to test, should go to
comp.sources.testers instead. A possible exception is test tool
vendors looking for beta testers.
If you are looking for a career in software testing,
comp.software.testing will welcome you. Just don't call it "beta
testing". Beta testing is typically conducted by end users of a
software product who are not paid a salary for their efforts.
------------------------------
Subject: 6. What other FAQs are relevant?
Danny Faught maintains the following FAQs, originally created by Brian
Marick:
Testing Contractors and Consultants List
Testing Courses List
Testing Tool Supplier List
These FAQs are available at <http://www.testingfaqs.org/>.
Please consult the appropriate FAQ before asking questions about the
information already contained therein. The Testing Tool Suppliers List
also contains a nice list of the available software test tools. If you
see anything that needs to be changed or added to these FAQs, please
contact Danny Faught at fau...@tejasconsulting.com so that they
may continue to be useful to everyone.
Since software testing is a subdiscipline of software engineering,
you will probably find the FAQs posted to comp.software-eng useful.
They can be found at <ftp://rtfm.mit.edu/pub/usenet/comp.software-eng/> or
<http://www.qucis.queensu.ca/Software-Engineering/>
The Problem Management Tools (bug tracking) FAQ is posted to
comp.software.config-mgmt, and is also available on news.answers and
the rtfm archives. The URL is
<http://www.daveeaton.com/scm/PMTools.html>.
Rick Hower maintains the "Software QA/Test Resource Center" at
<http://www.softwareqatest.com>. It covers a broad range of
questions about software testing.
Related to new trends, eXtreme Programming, here's the mini-FAQ:
<http://www.jera.com/techinfo/xpfaq.html>.
------------------------------
Subject: 7. What other resources are available?
The public comp.software.testing archives and the mailing list
gateway are no longer available. As an alternative, try Google groups at
<http://groups.google.com/groups?hl=en&lr=&safe=off&group=comp.software.testing>
Comp.software.testing articles are archived back to March, 1995.
If you're interested in quality measures (metrics), than
<news:comp.software.measurement>will interest you. However, I didn't
find any FAQ for this newsgroup.
If eXtreme testing is of interest, than
<http://www.xprogramming.com/software.htm> will interest you.
See also Ronald E. Jeffries's article at
<http://www.xprogramming.com/publications/SP99%20Extreme%20for%20Web.pdf>
http://www.cigital.com/hotlist/publications-ml.html>.
BetaSoft hosts Web forums on most popular testing tools and techniques
at <http://www.qaforums.com/>.
A hot topic is network security. Here are two pretty good references for testing.
<http://www.ideahamster.org/> <http://www.itprc.com/security.htm>
A quick reference explaining what is a "smoke test" can be found at
<http://www.construx.com/stevemcc/bp04.htm>.
------------------------------
Subject: 8. World Wide Web resources
Here are a few web sites that each try to connect you to much of the
software testing and quality information that is available on the web.
The Cigital Software Risk Management Hotlist
<http://www.cigital.com/hotlist/>
http://www.mtsu.edu/~storm>
http://www.soft.com/Institute/HotList/>
Kerry Zallar's software testing page
<http://www.testingstuff.com/>
Bret Pettichord's Software Testing Hotlist
<http://www.io.com/~wazmo/qa/>
StickyMinds.com Software Testing Area
<http://www.StickyMinds.com/>
Betasoft QA Links page
<http://www.QALinks.com>
Danny Faught's testing faqs page
<http://www.TestingFAQs.org>
Die Software Testing Website
<http://www.softwaretesting.de>
This website has lots of articles, book reviews, software testing related
software for download and a discussion forum. Most of the website is in
german language. All contents can be accessed free of charge.
LogiGear's QACity.Com : Resources for Busy Testers
<http://www.qacity.com/>
------------------------------
Subject: 9. What's the difference between QA and testing?
Sharon Codrington asks:
Please enlighten this young QA person as to the difference between
QA & Testing. Am I right in thinking that QA is more a preventive
thing, ensuring quality in the company and therefore the product
rather than just testing the product for software bugs?
And Bret Pettichord answers:
DING. DING. DING. You are correct.
Most testing jobs I see are nevertheless advertised as "QA". Some people
suggest that QC is a better set of initials to describe testing.
Don Mills writes:
In my courses and my practice, I stick to the ANSI/IEEE
definitions, which agree with what is generally held *outside* the
software arena. The definitions boil down to:
* TESTING means "quality control"
* QUALITY CONTROL measures the quality of a product
* QUALITY ASSURANCE measures the quality of processes used
to create a quality product.
Rick Hower addresses this question at
<http://www.softwareqatest.com/qatfaq1.html>.
------------------------------
Subject: 10. I'm looking for a test tool...
Did you read the Testing Tool Suppliers FAQ at
<http://www.testingfaqs.org/>? It has a list of many of the
available tools, plus a list of the commercial tools listings. While
every question about test tools isn't answered there, it's a good start,
and it'll at least allow you to ask a more directed question if you do
post a question about test tools.
When you do post a tools question, you should expect a few "check out
my company's tool" responses from a few vendors. This rarely gives
you any more information than is available in the tools FAQ, and which
vendors respond at any given time varies widely. Often you'll find a
vendor trying to drum up interest by posting the same thing in
response to every current article that mentions tools. Keep in mind
that your choices are much broader than those that happen to get
posted.
------------------------------
Subject: 11. How do I find information about testing object-oriented
programs?
Based on information originally provided by Brian Marick:
Books that include some discussion of testing OO programs:
Binder, _Testing Object-Oriented Systems_
Jacobson, _Object-Oriented Software Engineering_
Marick, _The Craft of Software Testing_
McGregor, _Object-Oriented Software Development_
Siegel, _Object Oriented Software Testing_
There are a vast number of articles about testing OO programs. I
suggest starting with the September 1994 Communications of the ACM,
which is devoted to this topic. Chase references. The February 1996
issue of Object has a piece on system testing with use cases.
Conferences like STAR and Quality Week and Pacific Northwest Software
Quality Conference seem to always have papers on this topic. I
suggest going to a conference and buttonholing people.
Web sites with discussion:
<http://www.cs.washington.edu/homes/gmurphy/testSTApp.html>
<http://www.testing.com/>
<http://www.toa.com>
Courses on testing object-oriented code (one by Robert Binder,
one by Ed Berard) are described in the Testing Courses FAQ. Also,
a new addition is an offering from Software Quality Engineering.
------------------------------
Subject: 12. How do I test web-related applications?
Rick Hower maintains a web page which includes a nice list of web
testing tools, and some guidelines for testing web sites.
<http://www.softwareqatest.com/qatweb1.html> Please contact him at
rick...@earthlink.net if you can offer additions or corrections to the
list.
One book that can be of help, "Testing Applications
on the Web" by Hung Q. Nguyen and "The Web Testing Handbook"
by Steven Splaine and Stefan P. Jaskiel.
One recent addition is Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid.
<http://www.StickyMinds.com/books.asp>
It also contains a Tool Evaluation appendix, comparing the
major tools (Mercury, Rational, Empirix, Compuware, Segue).
This matrix consists of approx. 30 pages of detailed
evaluation criteria.
------------------------------
Subject: 13. What is black box/white box testing?
Black-box and white-box are test design methods. Black-box test design
treats the system as a "black-box", so it doesn't explicitly use
knowledge of the internal structure. Black-box test design is usually
described as focusing on testing functional requirements. Synonyms for
black-box include: behavioral, functional, opaque-box, and
closed-box. White-box test design allows one to peek inside the "box",
and it focuses specifically on using internal knowledge of the software
to guide the selection of test data. Synonyms for white-box include:
structural, glass-box and clear-box.
While black-box and white-box are terms that are still in popular use,
many people prefer the terms "behavioral" and "structural". Behavioral
test design is slightly different from black-box test design because
the use of internal knowledge isn't strictly forbidden, but it's still
discouraged. In practice, it hasn't proven useful to use a single test
design method. One has to use a mixture of different methods so that
they aren't hindered by the limitations of a particular one. Some call
this "gray-box" or "translucent-box" test design, but others wish we'd
stop talking about boxes altogether.
It is important to understand that these methods are used during the
test design phase, and their influence is hard to see in the tests once
they're implemented. Note that any level of testing (unit testing,
system testing, etc.) can use any test design methods. Unit testing is
usually associated with structural test design, but this is because
testers usually don't have well-defined requirements at the unit level
to validate.
Definitions of these terms can be found in these references:
Beizer, _Black Box Testing_, p. 8.
Beizer, _Software Testing Techniques_, pp. 10-11.
Daich, et al., _STSC Software Test Technologies Report_, pp. 198, 206.
Kaner, et.al., _Testing Computer Software_, Second Edition, pp. 41-43.
Mosley, _The Handbook of MIS Application Software Testing_, pp. 68-76.
Myers, _The Art of Software Testing_, pp. 8-11.
Wilson, _Unix Test Tools and Benchmarks_, pp. 298, 306
------------------------------
Subject: 14. What are unit, component and integration testing?
The following definitions are from a posting by Boris Beizer on
the topic of "integration testing" in the c.s.t. newsgroup.
The definitions of integration tests are after Leung and White.
Note that the definitions of unit, component, integration, and
integration testing are recursive:
Unit. The smallest compilable component. A unit typically is the
work of one programmer (At least in principle). As defined, it does
not include any called sub-components (for procedural languages) or
communicating components in general.
Unit Testing: in unit testing called components (or communicating
components) are replaced with stubs, simulators, or trusted
components. Calling components are replaced with drivers or trusted
super-components. The unit is tested in isolation.
component: a unit is a component. The integration of one or more
components is a component.
Note: The reason for "one or more" as contrasted to "Two or
more" is to allow for components that call themselves
recursively.
component testing: the same as unit testing except that all stubs
and simulators are replaced with the real thing.
Two components (actually one or more) are said to be integrated when:
a. They have been compiled, linked, and loaded together.
b. They have successfully passed the integration tests at the
interface between them.
Thus, components A and B are integrated to create a new, larger,
component (A,B). Note that this does not conflict with the idea of
incremental integration -- it just means that A is a big component
and B, the component added, is a small one.
Integration testing: carrying out integration tests.
Integration tests (After Leung and White) for procedural languages.
This is easily generalized for OO languages by using the equivalent
constructs for message passing. In the following, the word "call"
is to be understood in the most general sense of a data flow and is
not restricted to just formal subroutine calls and returns -- for
example, passage of data through global data structures and/or the
use of pointers.
Let A and B be two components in which A calls B.
Let Ta be the component level tests of A
Let Tb be the component level tests of B
Tab The tests in A's suite that cause A to call B.
Tbsa The tests in B's suite for which it is possible to sensitize A
-- the inputs are to A, not B.
Tbsa + Tab == the integration test suite (+ = union).
Note: Sensitize is a technical term. It means inputs that will
cause a routine to go down a specified path. The inputs are to
A. Not every input to A will cause A to traverse a path in
which B is called. Tbsa is the set of tests which do cause A to
follow a path in which B is called. The outcome of the test of
B may or may not be affected.
There have been variations on these definitions, but the key point is
that it is pretty darn formal and there's a goodly hunk of testing
theory, especially as concerns integration testing, OO testing, and
regression testing, based on them.
As to the difference between integration testing and system testing.
System testing specifically goes after behaviors and bugs that are
properties of the entire system as distinct from properties
attributable to components (unless, of course, the component in
question is the entire system). Examples of system testing issues:
resource loss bugs, throughput bugs, performance, security, recovery,
transaction synchronization bugs (often misnamed "timing bugs").
------------------------------
Subject: 15. What's the difference between load and stress testing ?
Boris Beizer says:
> One of the most common, but unfortunate misuse of terminology
>is treating "load testing" and "stress testing" as synonymous. The
>consequence of this ignorant semantic abuse is usually that the system
>is neither properly "load tested" nor subjected to a meaningful stress
>test.
>
>1. Stress testing is subjecting a system to an unreasonable load
>while denying it the resources (e.g., RAM, disc, mips, interrupts,
>etc.) needed to process that load. The idea is to stress a system to
>the breaking point in order to find bugs that will make that break
>potentially harmful. The system is not expected to process the
>overload without adequate resources, but to behave (e.g., fail) in a
>decent manner (e.g., not corrupting or losing data). Bugs and failure
>modes discovered under stress testing may or may not be repaired
>depending on the application, the failure mode, consequences, etc.
>The load (incoming transaction stream) in stress testing is often
>deliberately distorted so as to force the system into resource
>depletion.
>
>2. Load testing is subjecting a system to a statistically
>representative (usually) load. The two main reasons for using such
>loads is in support of software reliability testing and in
>performance testing. The term "load testing" by itself is too vague
>and imprecise to warrant use. For example, do you mean representative
>load," "overload," "high load," etc. In performance testing, load is
>varied from a minimum (zero) to the maximum level the system can
>sustain without running out of resources or having, transactions
>suffer (application-specific) excessive delay.
>
>3. A third use of the term is as a test whose objective is to
>determine the maximum sustainable load the system can handle.
>In this usage, "load testing" is merely testing at the highest
>transaction arrival rate in performance testing.
------------------------------
Subject: 16. Should we discuss bug tracking tools in this newsgroup?
Discussing bug tracking tools is not within the original charter of
comp.software.testing. The comp.software.config-mgmt newsgroup seems
to have taken this topic under its wing. That group has a FAQ for
problem management tools. See the "What other FAQs are relevant?"
section for details.
Danny Faught maintains a FAQ under is wing for defect-tracking tools
at http://www.testingfaqs.org/t-track.htm.
------------------------------
Subject: 17. What kind of salary should a tester make?
Salary information is available at <http://www.pencom.com/industry.html>
and <http://www.computerjobs.com/salary_survey_search.cfm>. A nice
summary of the various surveys that are available is at
<http://www.ondaweb.com/sti/salary.htm>.
John Tyson writes:
> You could check the May '95 issue of Application Development Trends
> magazine. The article "Testing moves from purgatory to profession" by
> Linda Hayes [lgh...@metronet.com] has an excellent all-around
> (non-technical) article on testing and does include some regional
> salaries (albeit very general).
John also encourages testing consultants to report their rates at
Janet Ruhl's Computer Consultant's Resource Page -
<http://www.realrates.com>.
http://qualityprogress.asq.org/qp/nov97.html>
> to order a back issue.
Mark McWhinney writes:
> You might not want to use the survey results. While they are helpful
> for defining an organization's general pay scales, they are not very
> useful on a case by case basis. There are too many factors involved
> that may push a particular employee's base salary significantly above
> or below the average or outside the range.
>
> For QA and test people, these surveys are even more a problem. QA and
> test people are second-class citizens. They are paid less than their
> counterparts in development. In part this is due to the fact that QA
> and test people tend to be less senior and less experienced, therefore
> deserving of less pay. I have no problem with that. However,
> developers and testers with equivalent skill, education, and experience
> are not paid the same. This is wrong....
------------------------------
Subject: 18. Where can I find sample test plans?
See the following IEEE standards:
829-1983 IEEE Standard for Software Test Documentation
1008-1987 IEEE Standard for Software Unit Testing
1012-1986 IEEE Standard for Software Verification and Validation Plans
1059-1993 IEEE Guide for Software Verification and Validation Plans
You can contact the IEEE in the US at 800-678-4333. Also, Requisite
(800-732-4047) sells a Standards Pack with Microsoft Word Templates for
829, 830, and 1012 (part number 3100-05879). The Software Productivity
Center (<http://www.spc.ca>, 604-662-8181) sells Word templates,
including a test plan standard and a test plan template.
Several people recommend Kaner's book, _Testing Computer Software_, for
test plan samples. You'll find the ETET test plan in
<ftp://ftp.ecs.soton.ac.uk/pub/etet/etet1.10.3.tar.Z> under the path
"doc/testplan". See <http://www.acomtech.com/testplan.html> for an
adaptation of IEEE 829. And visit
<http://members.tripod.com/~bazman/frame.html> for a sample system test
plan. There is an outline of an IEEE based test plan on
<http://www.itc.nrcs.usda.gov/sqa/tst_outline.html>.
StickyMinds.com also have a template section for test documents at
<http://www.stickyminds.com/swtest.asp?btntopic=tem&Function=Search
&topcat=SWTST>.
Information in this section was gleaned from posts by: Burt Gearhart,
Wayne Woodruff, Brent Parsons, Castor Fu, Michele Mercer, Barry
Dorgan, and others.
------------------------------
Subject: 19. What is the best tester to developer ratio?
Reported tester:developer ratios range from 10:1 to 1:10.
Jeremy L. Mordkoff writes:
> There's no simple answer. It depends on so many things, I can't even
> list them all. Amount of reused code, number and type of interfaces,
> platform, quality goals, etc.
>
> It also can depend on the development model. The more specs, the less
> testers. The roles can play a big part also. Does QA own beta?
> Do you include process auditors or planning activities?
Boris Beizer adds:
> These figures can all vary very widely depending on how you define
> "tester" and "developer". In some organizations, a "tester" is anyone
> who happens to be testing software at the time -- such as their own. In
> other organizations, a "tester" is only a member of an independent test
> group.
>
> It is far, far, better to ask about the test labor content than it is
> to ask about the tester/developer ratio. The test labor content, across
> most applications is generally accepted as 50%, when people do honest
> accounting. For life-critical software, this can go up to 80%.
------------------------------
Subject: 20. Conferences
For further information about the conferences listed here and others,
see the Conferences section of the SRM Hotlist at
<http://www.cigital.com/hotlist/conferences.html>.
Opinions in this section are by Boris Beizer.
All of these conferences have published proceedings. Generally,
materials presented at these conferences and in their proceedings lead
the publication in regular journals (above) by about 2-3 years.
1. International Symposium on Software Testing and Analysis (ISSTA)
theory, academic, leading edge practitioners (see 3b in the
periodicals section.) Now transitioning to a summer conference that
will alternate with ICSE. Sponsored by ACM and ACM's SIGSOFT.
2. International Conference on Software Engineering (ICSE). Spring,
world-wide. Technical. Primary source after ISSTA for leading edge
results.
3. Quality Week (QW). Annual, San Francisco Bay Area. Biggest
Conference on Testing and QA. Typically 700+. Many vendors. Good
balance between technical/theoretical and practitioners. Very broad
base. Workshops. Sponsored by Software Research Institute,
q...@soft.com, 800-942-SOFT, 415-957-1441.
Quality Week Europe (QWE) is held in Brussels, Belgium in November.
4. Pacific Northwest Software Quality Conference (PNSQC). Annual,
Portland Oregon, October. Definitely not a regional conference,
despite the name. Very broad from technical/research to
practitioners. A shade more academic and leading edge participation
than quality week.
5. International Conference and Exposition on Testing Computer Software
(TCS). Less technical and more practitioner/management than Quality
Week. Smaller, 400+. Workshops. 17th annual conference was June
12-15, 2000, in Washington, D.C. Presented in cooperation with the
ACM SIGSoft, ASQ Software Division, IEEE Reliability Society, and
Software Technology Support Center.
6. Software Testing, Analysis, and Review (STAR). Software Quality
Engineering. STAR East is in Orlando in May, and STAR West is on
the east coast in the fall. About the same target as the
International Conference (5) above. Comparable level and interest.
Workshops. SQE 1-800-423-8378.
7. QAI International Software Testing Conference. More of a
tutorial/workshop than a conference. Newbie orientation. Fall,
Orlando, Florida. Quality Assurance Institute, 407-363-1111.
8. IEEE Compsac, world-wide, various locations. Fall. Typical IEEE
technical conference with a substantial number of papers of direct
interest to testing and QA. Not as many as IEEE Software
Engineering, though.
There are about a dozen other annual conferences with a substantial
testing content: Korea, Japan, Germany, to name a few. In addition,
there are about two-dozen "conferences", privately sponsored by various
individuals and organizations. These are not real conferences in that
there is no formal review process. Speakers are invited by the
conference organizer. Often, a heavy tutorial content. These are
usually small (under 100 participants, very few vendors). They range,
depending on the organizer, from superb to fair-value, to outright
rip-off. Sorry. I won't play 20 questions on this one.
(end of Beizer's comments)
Also of note:
International Conference on Software Quality (ICSQ), sponsored by
the Software Division of the ASQ.
International Symposium on Software Reliability Engineering (ISSRE),
sponsored by IEEE.
International Software Assurance Certification Conference (ISACC),
"The Software Risk Management Conference." ISACC is an
international conference in an annual series to be devoted
exclusively to the topic of software certification.
International Conferences on Practical Software Quality / Testing
Techniques (PSQT and PSTT). Two conferences a year, one in Minnesota
and one further south. Managed by Software Dimensions.
Software Testing Analysis & Review Europe (EuroSTAR).
Software Testing Analysis & Review Latin American (LatinSTAR)
in March 20-23, 2001.
The IFIP 14th International Conference on
Testing of Communicating Systems
Berlin, Germany, March, 19th - 22nd, 2002
http://www.fokus.gmd.de/events/testcom2002
------------------------------
Subject: 21. Periodicals
Comments in this section are by Boris Beizer
1. IEEE TSE (Transactions on Software Engineering). Monthly.
The most prestigious journal for testing stuff. Volume 1 number 1
published the landmark article by Goodenough and Gerhart. Almost
every issue since has had papers on testing and quality
assurance. I have all the issues back to V1 #1. Much of that
stuff is still relevant.
2. ACM TOSEM (Transactions on Software Engineering Methodology).
Quarterly. Relatively new journal (1992). Has prestigious
editorial board. Somewhat more theoretical than IEEE TSE.
3a. ACM SIGSOFT Software Engineering Notes. (from the Special Interest
Group on Software Engineering). Monthly. Goes back to May 1976.
This is the place where people publish stuff to stake out claims.
It is not as strictly refereed as IEEE TSE or TOSEM, but some
overview is provided to keep out the obvious crap. Important because
there is a two or three year waiting list for publication in IEEE SE
or ACM TOSEM. People who know this field read ACM SIGSOFT for
advanced information on what will be published "officially" in IEEE
SE and ACM TOSEM. You read an article and contact the author
directly for more information.
3b. ISSTA conference proceedings. Annual. (Known as TAV in the past --
and possibly new name changes in the offing.) ISSTA is the
theoretical/academic conference devoted to testing. This is the
most prestigious place to publish new results in testing theory and
software reliability theory.
4. IEEE Software. Six times a year. Rarely the latest stuff. More
like surveys and overviews once a subfield has become established.
Refereed, generally high standards. Mostly overviews, but
occasionally new stuff.
Additional comments from Scott Killops:
You might mention in your blurb about 'IEEE Software' the
regular "Quality Time" feature. This is wide ranging and
generally quite useful. I've often thought that a collection
of all of the "Quality Time" articles published to date would
make a worthwhile book.
5. Software Testing, Verification, and Reliability. Quarterly.
International journal published by John Wiley and Sons.
Prestigious international editorial board. High quality original
articles and excellent surveys. Comparable in scope to IEEE TSE
and ACM TOSEM, but sharply focused on testing, verification, and
reliability. ISSN 0960-0833.
<http://www.csc.liv.ac.uk/users/mrw/stvr.main.html>
<http://www.wiley.com/>
in...@qm.jwiley.com, cs-jo...@wiley.co.uk
John Wiley & Sons, Inc., Professional, Reference and Trade Group
605 Third Avenue, New York, NY 10158-0012
6. ACM Computing Surveys. Quarterly. Not specific to testing and QA,
but contains the most prestigious survey articles in the field,
typically only when a subfield is well established. The authors are
usually authorities. Articles are long and comprehensive. When an
ACM Survey on a topic appears, it usually means that the field has
matured to the point where it is possible to write meaningful books.
Volume 29, #4, December 1997, contains the most comprehensive survey
of testing research ever published--"Software Unit Test Coverage and
Adequacy", Hong Zhu, Patrick A. V. Hall, and John H. R. May, pages
366-427. The bibliography contains 221 entries. Anyone interested
in understanding the technical/research literature of software
testing should start with this survey.
7. Communications of the ACM. Monthly. Survey articles and overviews.
Sometimes (rarely) original stuff. More academic and
foundational/theory oriented than IEEE Software, but generally the
same level. Doesn't publish too much on testing.
8. Cutter IT Journal (formerly American Programmer). Monthly.
Cutter Information Corp., 37 Broadway, Suite 1, Arlingon MA
02474, phone 781-641-5118. Private journal with prestigious
editorial board. Non-technical, philosophical and overviews.
Management orientation. Essentially a good newsletter.
<http://www.cutter.com/itjournal>
9. Software Testing & Quality Engineering. Bi-monthly. Software
Quality Engineering, sqe...@sqe.com, 800-423-8378, fax
904-268-0733, phone 503-829-6806, <http://www.stqemagazine.com/>.
Formerly called Software QA magazine and Software QA Quarterly.
10. The Software Practitioner. Newsletter, 6 times a year.
Computing Trends, 1416 Sare Rd., Bloomington IN 47401. Robert
L. Glass - Editor-in-Chief, rlg...@acm.org. Non technical,
management, overviews. Not too much on testing.
http://SoftwarePractitioner.com/
11. Crosstalk: The Journal of Defense Software Engineering. 6 times a
year, free. Software Technology Support Center, Ogden AFB UT 84056.
801-777-8057. Heavy DoD orientation. Practitioner/Management/
Philosophical. Broad based on software engineering, including IV&V.
http://www.stsc.hill.af.mil/CrossTalk/crostalk.html>
cust...@software.hill.af.mil.
12. Other journals. I'm sure I've offended several editors. There are
journals concerned with testing, QA, and software engineering
published in Japan, Korea, Germany, and Australia. Other journals,
such as IEEE Transactions on Computers, IEEE Transactions on
Reliability, ACM SIGPLAN, ACM SIGARCH, ACM SIGSIM, ACM SIGMETRICS,
ACM OOPSLA, all publish material of interest to testing from time to
time, especially insofar as hardware testing results and theory
apply to software testing; also language issues vis-a-vis testing.
(end of Beizer's comments)
The American Society for Quality has begun publication of Software
Quality Professional. This journal will span the
body of knowledge for the Certified Software Quality Engineer.
<http://www.asq.org/pub/>
Commercial newsletters - the newsletters below are available for free to
potential customers and some provide very good technical content.
Quality Techniques Newsletter (QTN), formerly Testing
Techniques Newsletter (TTN). Monthly, in electronic
form only. Software Research, Inc., 415-957-1441,
800-942-SOFT. Send "subscribe" in the body of a message
to q...@soft.com.
<http://www.soft.com/News/QTN-Online/>
The Payne Report, http://www.cigital.com/paynereport/
Tejas Software Consulting Newsletter, emailed monthly. See
<http://tejasconsulting.com/#news> for archives and to get a free
subscription.
------------------------------
Subject: 22. Books
Books relevant to software testers are listed in the Books section of
the SRM Hotlist at <http://www.cigital.com/hotlist/publications-books.html>.
See also <http://www.StickyMinds.com/> under "Books" for Software
Testing and Software Quality Engineering's book list, formerly
known as the SingleSource Guide. See the misc.technical.books FAQ
<http://www.faqs.org/faqs/books/technical/> for contact
information for technical book publishers. Comp.software.testing readers
have recommended Reiter's Scientific and Professional Books
<http://www.reiters.com> and Fatbrain.com <http://www.fatbrain.com/>.
------------------------------
Subject: 23. Bibliographic resources
Comments by Boris Beizer.
1. ACM Computing Reviews. Monthly. The primary review journal. Most
good papers on testing and quality assurance are eventually reviewed
here. Extensive evaluations and criticisms by excellent reviewer
pool. <http://www.acm.org/reviews/>
http://www.acm.org/catalog/journals/107.html>
Also, some books have large bibliographies that are very useful, such as
those in Software Testing Techniques and Testing Computer Software.
------------------------------
Subject: 24. Organizations
The organizations below are devoted to software testing or quality, or
have special interest groups with such a focus. National and
International organziations are listed first, followed by local
organizations.
National/International
----------------------
American Society for Quality
611 E. Wisconsin Ave.
P.O. Box 3005
Milwaukee, WI 53201-3005
800-248-1946, 414-272-8575
<http://www.asq.org>
The ASQ (formerly ASQC) administers the CQE (Certified Quality
Engineer) and CSQE (Certifified Software Quality Engineer)
designations.
The ASQ press has many titles that may be of interest to software
testers. Stolen from a conference announcement:
> Founded in 1946, the American Society for Quality (ASQ)
> provides a variety of professional, educational, and informational
> programs reflecting the changing needs of business and industry.
> Headquartered in Milwaukee, WI, ASQ has been the leading quality
> improvement organization in the United States for almost 50 years.
> The Society is composed of more than 135,000 individual members and
> 1,000 sustaining members worldwide. ASQ's vision is to be the
> world's recognized champion and leading authority on all issues
> related to quality. The mission of ASQ is to facilitate
> continuous improvement and increased customer satisfaction by
> identifying, communicating, and promoting the use of quality
> principles, concepts, and technologies.
ASQ's Software Division
> ASQ's Software Division is comprised of [sic] more than 5,000 members
> including software quality professionals and software engineers
> interested in applying quality principles to the field of software
> development. The Division develops a software engineer
> certification program, publishes a quarterly newsletter, works with
> the Software Quality System Registration Committee on establishing
> an ISO 9000 Software Registration Program in the United States,
> interacts with other professional software organizations such as
> the IEEE and the ACM (Association for Computing Machinery), and is
> involved in many other activities.
<http://www.asq.org/about/divtech/softdiv/swqweb.html>
October is National Quality Month (in the U.S.), as decreed by
Ronald Reagan in 1984.
Quality Assurance Institute
7575 Dr. Phillips Blvd., Suite 350
Orlando Florida 32819
Phone: 407-363-1111
Fax: 407-363-1112
<http://www.qaiusa.com/>
The QAI publishes The Journal of the Quality Assurance Institute
monthly.
They provide CQA certification. Steve Driscoll wrote:
> The CQA is someone who has demonstrated a mastery of the Common
> Body of Knowledge(CBOK) for ensuring the quality of systems (which
> often include software). While ISO-9000 is considered part of the
> CBOK, it's actually independent of the CQA designation (there is a
> certification program for ISO auditors).
> ... while my life is still pretty much the same as it was prior to
> obtaining the CQA designation, I found that the reading and
> studying to prepare for the exam extremely beneficial in my duties
> as a Quality Analyst. I recommend it to anyone who is interested
> in producing quality software (which includes developers as well
> as quality analysts).
Society for Software Quality
> Vision - To be recognized as the Society for those interested
> in promoting "quality" as a universal goal for software.
>
> Mission - Promote "quality" as a desirable attribute of software.
PO Box 86958
San Diego, CA 92138-6958
619-297-1544
<http://www.ssq.org/>
Software Process Improvement Network
> The Software Process Improvement Network is comprised of
> individuals who want to improve software engineering practice. The
> individuals are organized into regional groups called "SPINS" that
> meet and share their experiences initiating and sustaining software
> process improvement programs. They meet annually at the SEPG
> (Software Engineering Process Group) Conference, which is
> co-sponsored by the SEI and a regional SPIN.
(from the web page)
<http://www.sei.cmu.edu/collaborating/spins/spins.html>
British Computer Society
Software Testing BCS Specialist Group
> The Group has a number of full-day meetings per annum and is host
> to the international conference on software testing - EuroSTAR. It
> is producing a standard on component testing and has a sub-group on
> Statistical Testing Methods. A full library of testing-related
> material is available at meetings.
>
> It promotes awareness of testing, competence in the field, and
> encourages research.
(from the web page)
<http://www.bcs.org.uk/siggroup/sg55.htm>
Chinese Association for Software Quality (CASQ)
This association is formed to share information and experiences
amongst its members and people on software quality engineering.
<http://www.casq.org>
ca...@newcastle.ac.uk
Swedish Association for Software Testing (SAST)
SAST is a non-profit organisation with members from all different
companies that are interested in Software Testing within Sweden.
The purpose of SAST to improve status of Software Testers and
Software Testing, through informal education and exchange of
information. Since this is a Swedish organization most activities
are in Swedish. If you are interested you are welcome to contact
Lise-Lotte Karlsson Boman (lise-lotte.k...@posten.se) or
Jan Warg (j...@warg.se).
<http://public-www.pi.se/~sast>
TestNet
TestNet is a Special Interest Group in Software Testing with over 400
members of over 150 companies and organisations mainly in the Netherlands
(although there are a few members from Belgium). The meetings and
news-items are in Dutch.
> The purpose of TestNet is the professionalization of testing IT products,
> and an increase in the awareness and the importance of testing as a
> profession in its own right. TestNet stimulates the exchange of
> professional knowledge and practical experience amongst testers, and
> stimulates research, from a scientific standpoint as well as from a
> practical perspective.
(from the web page)
If interested or for information contact at secre...@testnet.org
<http://www.testnet.org>
tes...@testnet.org
CRIM Software Test Centre
CRIM is a transfer and liaison centre with fifteen years'
experience in leading-edge information technologies and
computer application sectors.
CRIM Software Test Centre (STC) helps big or small businesses
and other kinds of organizations boost product quality, cut
risk and optimize investments. Technologically, CRIM STC
pinpoints the strengths and weaknesses of software applications
based on pre-established criteria. CRIM STC may also recommend
various ways of improving and optimizing the product testing
and development processes.
<http://www.crim.ca/en/index.epl?selec=a000&href=/en/ctl/index.htm>
Local organizations
-------------------
Most of the organizations above sponsor local organizations. Contact them
to find local organizations near you. In this section you'll find local
organizations that don't appear to have a national affiliation.
The Software Quality Institute At UT
(stolen from a conference announcement)
> The Software Quality Institute (SQI) at The University of Texas at
> Austin is a multidisciplinary partnership between UT-Austin and the
> software and information systems industries in Texas. It is
> recognized as a leading authority on and champion for software
> quality. SQI was founded in 1993 for the purpose of strengthening
> organizations to compete more successfully in global markets
> through sponsorship of seminars and conferences. An advisory group
> of 24 industry and academic representatives guides the Institute.
> Its popular programs include: a 13 week long Software Project
> Management Certificate Program, one- and two-day seminars offered
> to the public and as in-company contract courses, our "Software
> Quality Matters" quarterly newsletter that addresses issues of
> concern relative to quality issues, sponsorship of monthly forums
> for discussion of software issues (including the Software Process
> Improvement Network (A-SPIN), the Austin Software Executives' Group
> (ASEG), and the Austin Forum for Object-Oriented Technology
> (AFOOT), and on-line services which include a Worldwide Web
> Home Page and topical newsgroups.
The web page is <http://www.utexas.edu/coe/sqi/>. The newsgroups
seem to be reachable only at UT. For information on newsletter
subscriptions, contact in...@sqi.utexas.edu.
Central Ohio Quality Assurance Association
Mission Statement: To provide and promote continuous quality
improvement information to software organizations in our
communities. COQAA is organized to share state-of-the-art quality
assurance methods, tools, and techniques among its members.
P.O. Box 14191, Columbus, Ohio 43214-0191
contact: Jerry Fehribach, 614-447-3600, jfehr...@cas.org
<http://www.coqaa.org>
Australian Software Quality Research Institute
Griffith University, Brisbane, Australia
<http://www-sqi.cit.gu.edu.au/>
Triangle Information Systems Quality Association (North Carolina)
http://www.rtpnet.org/tisqa/
contacts - Dennis Brandel 72652...@compuserve.com
David Wood DAVD...@VNET.IBM.COM
North Carolina Quality Leadership Foundation
800-207-5685
http://www.rtpnet.org/ncqlf/
New England Software Quality Assurance Forum (NESQAF)
NESQAF is a regional non-profit society created by software quality
professionals for software quality professionals. Each month,
professionals from software companies and MIS organizations along
with other industry experts exchange ideas, principles, and
experiences on the latest quality assurance techniques and
technologies. An email distribution list is available.
Monthly meetings in Cambridge, MA.
Contact - Alan Titelbaum, alan.ti...@tfn.com, 617-856-2467.
<http://www.nesqaf.org/>
Software Testing Institute
Offers industry research, surveys, publications, online services
and seminars to software development and testing professionals. A
list of software testing product vendors, including web site and
e-mail links, is available from our web site:
contact - Susan Archer, Director (sar...@metronet.com)
726 Loganwood Ave., Richardson, TX 75080
972-680-8507, fax 972-680-8905
Centre for Software Reliability
The Centre for Software Reliability (CSR) is a research centre
within the Department of Computing Science at the University of
Newcastle upon Tyne; it conducts research on how to achieve
improved levels of dependability from computing systems.
c...@newcastle.ac.uk
<http://www.csr.ncl.ac.uk/>
Seattle Area Software Quality Assurance Group (SASQAG)
Purpose: To Promote Professional Software Quality Practices
<http://www.sasqag.org/>
------------------------------
Subject: 25. Contributors
Thanks go to the people who have contributed to the FAQ in various
ways, including the team of assistant FAQ maintainers - Chris Petrov,
and Richard de Graaf, and Danny Faught. Email addresses for some of
those mentioned herein are listed below.
Boris Beizer - bbe...@acm.org
Bob Binder - rbi...@rbsc.com
Richard de Graaf - r.dg...@chello.nl
Barry Dorgan - baz...@bigfoot.com
Steve Driscoll - sdri...@oclc.org
Sigrid Eldh - Sigri...@ehpt.com
Danny Faught - fau...@tejasconsulting.com
Rick Hower - rick...@earthlink.net
Scott Killops - Scott_B...@ccm.jf.intel.com
Jennifer Larsen - Jennife...@datatel.com
Brian Marick - mar...@testing.com
Mark McWhinney - m...@powertest.com
Don Mills - don...@voyager.co.nz
Jeremy L. Mordkoff - mord...@acm.org
Charles Nichols - nic...@cse.ogi.edu
Chris Petrov - petro...@usa.net
Bret Pettichord - br...@pettichord.com
Raymond Rivest - rri...@crim.ca
Laura Rose - la...@PACorp.com
John Tyson - j...@acm.org