Beizer Bug Taxonomy and Books
Boris Beizer
domain at the time of publishing. For those who are interested, I have
added it to this posting. I don't do attachments. You can use this and
reproduce it as per the conditions in the document.
2. Book Availability:
You may have heard that various of my books are out of print. This
is incorrect. Those that are not available from the publisher are available
from me.
1. Black Box Testing -- Wiley --available from publisher and book stores.
2. Software Testing Techniques, 2nd Edition -- $45 from author
3. Software System Testing and Quality Assurance -- $45 from author
4. Software Quality Reflections $40 -- from author.
The last is my collection of essays and poems (e.g, "The Running of 303", "
The Maven") that have appeared in various conference proceedings and on the
Software Research Web page and TTN Newsletter
Terms: Includes shipping by surface mail, book rate, within the US. For
Non-Us shipping and express mail, send me an email for a final quote.
Payment by money order, in advance, no returns. Foreign money order by
draft on US bank, or corresponding bank in US $ only.
---------------------------------------------------------------
.Table of contents.
----------------------------
-------------------------------------
Software Quality Reflections
Essays, Dialogues, and Poems by
Boris Beizer, Ph.D.
Copyright, 2000 by Boris Beizer
No part of this document may be reproduced or converted to any other form by
any electronic, manual, and/or mechanical means, including but not limited
to: photocopy, recording, taping, facsimile, transmission, scanning, storage
in a computer and/or memory--without the written permission of the author.
This document is provided for information. The material therein remains the
sole property of the author who retains all beneficial rights thereto. Reg.
# TX 5-245-789 6/16/2000
Printed in the United States of America
Boris Beizer
1232 Glenbrook Road
Huntingdon Valley, PA 19006 USA
16 15 14 13 12 11 10
Library of Congress Cataloging-in-Publication Data
Beizer, Boris, 1934 -
Software Reflections.
1. Computer software-Testing. I. Title
Preface
I got into the computer business in 1956 and full-time into software in
1962 - almost 40 years. From 1957 on I started writing and publishing pieces
in various computer journals. In 1985 I began regular participation in
conferences concerned with software quality and testing. In support of those
conferences, where I usually gave a talk, I prepared an essay on some aspect
of software quality. By 1999 I had accumulated well over 100 of these
articles and conference talks. Throughout this career I also wrote many
poems, some of which were published and some of which accrued a measure of
notoriety. Since 19965 I have been posting mini-essays to news groups and
discussion groups devoted to software quality issues, such as
comp.software.testing. Most of the mini-essays in this collection stem from
such postings.
People continually ask me for reprints of the above works-most of which are
no longer available from the original publisher. In response to these
requests I have assembled this selection of the essays, dialogues, and poems
which are still germane today; and omitted the two-thirds in which my
predictions were wildly incorrect, of course, or those that do not appear to
hold useful lessons for software quality as it is today. I made few changes
to them-the minimum required to bring them up to date. Where appropriate, I
have inserted prefatory notes to set the context in which the piece was
first written.
Unlike a typical book, in which redundancy is to be avoided, there is some
overlap in these essays. I thought about removing the redundancies but
decided against it. Each piece, as now presented, can stand alone - which
would not be the case if I removed the redundancies. The advantage to the
reader is that you can read it in the short bursts which the present
frenetic pace of software development seems to limit you. Furthermore, the
fact that a subject is discussed in more than one essay provides different
slices of the subject and therefore different perspectives.
Having explained away the redundancies in this book, let me address the
inconsistencies and contradictions-they're there, I freely acknowledge them,
and I hope that I have found them all and left them in deliberately. I could
take refuge in Ralph Waldo Emerson's pithy observation: "A foolish
consistency is the hobgoblin of little minds, adored by little statesmen and
philosophers and divines." But that won't do. The seeming contradictions
reflect the immaturity and the complexity of our business. Some of the
apparent contradictions have to do with where on the maturity scale you are.
What is correct in one state is wrong in another and vice versa. So in some
essays you'll see strong arguments for independent testing and in others,
arguments against it. You'll see me providing arguments as to why all
metrics are probably flawed or even impossible, and then strongly urging
that you implement an aggressive metrics program. It is not my purpose to
play mind games with you-but to show you different aspects of software
quality and to arm you against simplistic yes/no formulas and rigid
rules-which we haven't yet worked out.
Boris Beizer
Huntingdon Valley, PA, June 6, 2000
The items in italics are humorous and/or poems.
Contents
The Running of 303 1
The Four Horsemen 5
The New Model Process 17
Ten Axioms of Imperfection 26
What Constitutes Minimum Testing 27
Software Development Process-What's Really Important 29
How I used 2100 Memory Locations and Overflowed 33
Software Is Different 35
What is a Quality Culture? 55
The Tester's Soliloquy 58
Quality Maxims 59
Can Software Engineering Be Like Other Engineering 63
Songs of Automation VI - The Smith 74
Of Models and Abstractions 75
The Black-Box Vampire 89
Can Test Techniques Effectiveness be Quantified? 103
Reflections in a Carafe of Rosé 117
Making Software Testing More Effective 119
Testing is Obsolete 131
Quality is not the Goal 143
Quality is to Delight 153
Fan Letters I didn't Answer Department 160
The Walk Around 161
Favorite Conference Evaluations 168
Testing Big Systems 169
A Revolutionary Proposal 172
The Pepperoni Pizza Paradigm 173
BLANK 178
Testing, Seven Virtues, Seven Sins 179
Testing, Best and Worst Practices 191
Unbanning The Bug 199
What Makes a Good Tester? A Baker's Dozen 203
Real-Time Software Testing? 209
Respice, Prospice, Adspice 213
Turing Machines 242
ISO 9000 243
In Each of Us A Monster Dwells 245
Software Engineering is Illegal 255
The International Scene 267
Growing Pains 269
On Being Competitive 271
Misk Ramagement 101 287
Shibboleth 302
Losing It 303
Jet Lag 310
Y2K and All of That 311
Sexism in Testing 323
Test Automation Issues 331
Letter to Prometheus 337
Good Enough Software 339
Tokens 344
Retrospective on Software from 2011 345
The Mavin 365
Bibliography 369
Index 377
_________________________________
I think indexes give a good a good idea of content. So here it is:
Index
ABET 255, 257, 266
abstraction 75, 83, 84, 87, 221
ACM 259, 264, 266
age 278
aggressiveness 329
America's Cup 303, 304
analysis 64, 141, 142
anomaly tracking 233, 234
application 204, 205, 305, 306, 325, 349, 360
Art of Software Testing 93, 217
artifact 64, 66
assumptions 68
ATEC 223
audits 214
auto manufacturer 339, 340
automation 122, 126, 127, 132, 133, 143, 148, 193,
194, 224, 328, 331-336, 354, 356, 361
axioms of imperfection 26, 107
Baldrige Award 152, 287, 288
benchmark 152
best practice 235
beta testing 92, 187, 188, 198
big system 169-171
books 164, 165
brown bag sessions 163, 283
bug 1, 60, 61, 108, 144, 199-202
affect 60, 61
classification 233, 234
cluster 222
density 57
distribution 222
-feature interaction 170, 333
-free myth 120, 127, 144, 280, 339
guilt 339
,integration 94, 191
,interaction 297, 299
,interlock 195
,nondeterministic 219
prevention 55, 139, 214, 290
priority 195
reporting 286
,resource loss 170, 192, 195
severity 298
,static 209, 210
statistics 181, 182
symptom 60, 61
-symptom mapping 299
,synchronization 170, 192, 195, 209, 211, 212
,synergistic 40, 41
,system 94, 194
,test execution 332, 333
,timing 170, 195, 209, 212, 219
,tolerance to 339
,unit 94, 191
build process 240
bulletin board 163
bulletproof software 285
Bureau of Labor Statistics 15
C, C++ 222, 346
cache memory 210
cafeteria 167
capital investment 20
capture/playback 132, 194, 228, 229, 236, 240, 283,
325, 335, 353
CASE 133, 214
CAST 133
causality 300
CCITT 127
CD-ROM 9, 10, 157, 158, 285
chaotic system 71, 72
Cleanroom 214-216, 294
clipper ship 276
CMM 152
COBOL 19, 214, 222, 276, 346
COCOMO 56, 282
code ignorance 101
code-based test generator 229, 230
combinatorial test generator 227, 228, 237
community 274, 275, 279
compatibility testing 147, 350, 354, 359, 360
competition 121, 145, 272, 274- 279, 293
compiler 94, 184
complexity 41-44, 95, 214, 215
,algorithmic 43
,architectural 43
,combinatorial 333
escalation 333
-functionality inversion 42-44
growth 68
,proportional 42
,square law 334
component technology 9, 169, 191
testing 134, 169, 192, 224
composition, Principle of 45, 46, 68
computer industry 6
consensus seeking 329
conference room 163
confidence, human 297
configuration 3
testing 28, 94, 186, 187, 354, 357, 36
control 21, 23, 31, 56, 138, 139, 194, 334
consolidation 7, 8, 19, 238, 339
consultant 264
continual improvement 288, 289
control-flow testing 135
cost 25
-benefit analysis 143, 148, 149, 290
consequential 299
prediction 56
cover, coverage 97, 353
branch 55, 135, 191, 216, 222, 267
criteria 341, 342
, data-flow 223, 236
predicate 55, 191
standard 222
statement 27, 191
testing 180
tool 27, 94, 101, 172, 216, 221, 222,
236, 240, 341
CPU speed 210
creativity 24, 307, 308, 329,352 ,363
Cromwell, Oliver 23-25
crossed transaction 35, 36
cubicle 165-167
customer service 284, 285
data 59
sharing 293
data flow
diagram 214
testing 135, 136
debugging 140, 213, 217, 219
decision table 19, 214
decomposition 46, 47, 68
defensive code 184, 185
deinstallation 10, 158
dependability 221
design 32, 86
model 86
restriction 52
validation 32
Detroit mentality 304, 305
developer 87,166, 172
disc optimizer 210
discipline 24
displacement 13, 274, 276, 277
distributed software 219, 337, 338
diversity 166, 167
documentation 162, 163, 214
cost 61, 62
workaround 59
DoD 90, 91, 259
domain testing 99, 136
downsizing 7
education 14, 15, 282, 283, 286
efficiency 18
Email 271, 272
embedded software 210, 211
empiricism 64
employment 12, 13, 15, 16
encapsulation 52
engineer 255, 263
engineering 63, 64, 70, 72, 83
process 290, 291
software 65, 262, 263
entry criteria 31
epidemiology 294
equipment 166
Europe 267, 268
exit criteria 31
experimental blindness 294
extreme-value testing 180, 181
FAA 90, 259
failure 221
probability 57, 299
fair play 306, 307
false confidence 114, 147, 181, 182, 187, 222
feature 145
-driven testing 17 9
interaction 170, 333
wars 280
finite-state
behavior 211
testing 99
flowcharts 214
Flying Scot Inc 153, 154
forgiveness 152
formal specifications 215
functional decomposition 214, 215
genetic algorithms 220, 221
global data restriction 52
good enough software 340
GUI testing 101
hardware 6, 20, 305, 348
compiled 275
high technology 303, 304
honesty 54
hyperbabble 158
IBM 8,18, 92, 205, 222, 223
IEEE 127, 243, 244, 259, 266
ignorance 278
Imhotep 102
incremental improvement 80
independent testing 230, 231, 240, 241, 354
India 8
industry
displacement 13
maturation 8, 17
infrastructure
software 351, 354
testing 179, 180, 193
initial state 103, 104
innovation 16, 268, 274, 275
input vector 103
inspection 23, 55, 214, 215, 216, 341
installation 10, 138, 158, 282
instruction manuals 285
instrumentation 11, 55
integration 32, 169, 363
bug 94
plan 240
testing 27, 28, 134, 140, 169, 170, 191,
192, 209, 211, 224, 225, 236, 240, 354, 355
intellectual control 44
interface testing 146, 192
international 267, 268
outsourcing 8
Internet 6
inverse profile testing 212
ISO 9000 243, 244, 279
Japan 267, 268
JAVA 222, 338
journals, technical 164, 165
knowledge, adequate 50, 51, 68, 70-73
labor content 331
language barrier 306
lawyer 293, 339, 342, 343
liability 293
librarian 164, 165
library 164, 165
life-cycle 23
litigation fear 119, 120, 293
locality
principle of 38-40, 68
severity 39, 40
space 38
time 38, 39
localization testing 354, 355
love 273, 274
mainframe 6, 17, 350
maintenance 12, 21, 22, 138, 331, 332, 363
,corrective 332
,progressive 332
research 140 ,219
testing 191
management 17, 22, 25
manual
regression testing
testing 193, 195, 196, 225, 229, 331, 332, 334
testing error 332
manufacturing process 290, 291
marketing 13
maturation 28
maximum disgust, principle of 59
meetings 163, 164
megaproject 7
memory monitor 237
mentat 275
metrics 30, 56, 163, 232, 233, 241
Microsoft 6, 8, 12, 147, 235, 276, 281, 339, 349
model 26, 64, 75, 76, 78, 82, 85, 86, 88, 98, 102, 216
,behavioral 98-102
,best 102
,concrete 85
,detail 78
-driven test generator 229
,mathematical 64, 83-85
,mental 102
,reliability 342
,structural 98-102
modularity 19, 214
MS-DOS 276
multimedia 158
mutation analysis 114, 115, 219, 237 ,241
n-version programming 87, 214, 215
NCEE 255, 257, 261, 266
neatness 166, 167
NIST 15, 129
Novell 8, 147
NSPE 257, 266
object-oriented programming 7, 9,43, 169, 191, 215,
220, 225, 236, 354, 356
objectivity 91, 94, 216, 230, 231, 341
OLE 285
open systems 349
operating system 210, 348
optimization 67
oracle 104
output vector 103, 104
outsourcing 7, 8, 238
overtesting 127, 343
para-professional 357
paradigm 52, 190
pareto principle 27, 222
PC 6, 17, 19, 92, 146
Pepperoni-Pizza Paradigm 173-177
perfect software myth 120, 127, 144, 280, 339
performance 18, 149-151, 184
testing 170, 196, 197
permanence 272-275, 283
permission 152
pesticide paradox 112, 116, 181, 182, 188, 195, 292
Petri Nets 212
physical science 64
physical system 36, 37
pink profession 325, 326
piracy 285
post-industrial society 7
predicate cover 222, 239
pricing 280
probable correctness 342
process 29-32, 55, 214, 215
control 30
document 29
experiment 293, 294
training 29
product goals 282
productivity 9, 16, 331
professional engineer 256, 259, 261, 262, 264, 265
professionalism 20, 324
profiler tool 236
programmer 183, 185, 186, 190, 357, 363
guilt 200
immunity 188
myths 212
objectivity 94
qualification 56
subjectivity 91
tricks . 212
unemployment 346
programming 32, 90, 203, 204, 363
language 94, 214
projection 86, 105
proof of correctness 214-216
prophecy 5
prototype 84, 85, 94, 137, 138, 189, 231, 353
publisher 13
qualification test 12
quality 47, 48
assessment 161-168
assurance 21, 23, 24, 35, 55, 137, 221, 241
championship 235
control 55, 342, 343
culture 12, 55-57, 161
metric 48, 49
movement 155, 156, 287, 288
myths 162
objective 57, 143, 144, 331, 339, 340
quantification 30, 64, 67
metric 49, 50
,principle of 68
random testing 226, 227
real-time software 209-212
realism 79-81, 83, 84, 85, 179, 183, 211, 224
recovery 60
regression testing 195, 196, 219, 231, 232, 237, 240
manual 332
rehost 19, 20, 23
reliability 282
model 342
testing 196 ,234
replacement market 5
requirement 31, 55, 91, 94, 137, 138
testing 192, 193
traceability 31
validation 31
research 283, 301, 342, 343
responsibility 24
retrospective analysis 115, 116
reuse 9, 19, 20
reviews 55, 214
risk
management 287, 289, 293
model 171, 295, 298-300
,perception of 308
policy 299
profile
Romans 307, 308
safety analysis 219
safety limits 44
saturation 6
scale effects 296, 299
schedule 25, 97, 151, 152
fiction 282
prediction 56
scripting language 225, 226, 236, 237, 240
SEI 152
self-deception 211
semantic type restriction 52
service 350, 359
companies 350, 361-363
cost 280
industry 13
policy 11, 159
sexism 323-330
shelfware 235
shrink-wrapped software 7, 8, 94, 95, 238, 239, 272
simulator, 94, 211, 212
slicing 219
social skills 56, 329
software 65, 67, 275, 276, 305
adaptive 11, 61, 158, 284
author 350-353, 363
buyer 357
complexity 41-44, 95
component 239
consumer 339
cost 172, 281
development 8, 53, 54, 210, 211, 214
distributor 350, 356-361
employment 8, 11
engineering 5, 36, 52, 63, 65, 68, 73, 217,
255, 257-259, 261 262, 263, 266
Engineering Conference 259
Engineering Institute 259
environment 299
evolution 283, 284
gouging 349
-induced stress 60
industry 13, 345, 346, 350, 351, 363
,infrastructure 296, 297, 299
objective 61
psychology 60
publisher 350, 354-356
redundant 8
reliability 183, 220, 295, 296
retail vendor 10
service company 351, 356
technician 10, 11, 14, 357, 358, 363
utility of 69
specification 23, 59, 91
specification language 214
specification-based test generator 335, 336
standard 127-129, 243, 244, 279, 280
unit testing 27
statement cover 222
statistical
confidence 297, 298
inference 300
quality control 289, 291, 292
validity 300
statistics, naive 300
stochastic testing 212, 227, 299
stock market 293
stress testing 194, 195, 212
strong typing restriction 52
structured software 19, 52, 214
stub 214
style restriction 52
subsumption hierarchy 116, 218, 219
symbolic execution 214, 215
synchronization 211
synergistic bug 40
syntax-driven test generator 228
synthesis 64, 141
system
architecture 348, 349
bug 94, 194
testing 28, 134, 135, 169, 170, 191, 192,
219, 222, 223, 236, 354
technology 5, 12, 16, 234-237, 275, 282
temporal logic 212, 219, 221
terminology 121, 122
test, testing 12, 20, 21, 24, 32, 86, 90, 91,
97, 102, 104, 211, 213, 215, 216, 21 8, 363,
,alternatives to? 214, 215
architecture 180
,behavioral 134, 170, 212, 360
best practices 191
black-box 89, 90
bug reporting 335
case 103
clear-box 89
code-ignorant 92
configuration control 230, 240, 334
cross-platform 28
deinstallation 28
design 97, 98, 101, 325
design automation 126, 127, 132, 133,194,
331, 334, 335, 354, 356
-developer relation 329, 330
driver 101, 225, 237, 240, 335
dynamic 209, 212
environment 123
equivalency 332
execution 98, 325
execution automation 132, 193, 194, 224, 331,
334, 335 356
faked result 208
feature-driven 179
generator 102, 194, 226-230, 237, 240, 325
glass-box 89
gray-box 89
harness 354
history 131
Independent 94, 122, 123, 185, 197, 237
inspection 208
installation 28
interface 186
Inter-operability 146, 147
labor 12, 213, 354
library 22
literature 216-218
localization 28, 94
maintenance 333, 334, 361
management system 237
methods 100
model 86
myths 95, 96, 328
negative 56, 91
network 28, 94
objective 289, 290
performance 94
permanent 333
platform compatibility 94
positive 56, 97
practice 137-140, 218
purpose of 96, 97, 101
real-time software 209-212
research 140, 141
recovery 28
redundant 94
security 28, 94
service 124, 125
specification 105
state-box 89
static 209 212
structural 134
suite 353, 355
synchronization 211
taxonomy 219
technique 55, 97, 105-107, 136, 137,218, 219, 240
technique effectiveness 103, 106, 108-111,
114, 119
technology 95, 132, 218, 221-234, 237, 238
theory 217-221
timing 211
tool 123, 124, 203
training 21
usability 28, 94
value-added 94 197
white-box 89
testability 18, 139, 140, 171, 220, 342
analysis 241
model 290
research 141
tester 87, 166, 203-208
application knowledge 204, 205
chaos tolerance 205, 206
credibility 204
cunning 208
entry-level 204
honesty 208
hyper-sensitivity 205
intelligence 205
know-how 204
loser image 204
non-professional 323, 326
organization 207
qualification 56, 213
salary 90
skepticism 207
social skills 206
status 327, 328
talent 328
techology 208
tenacity 206, 207
toughness 207, 208
women 206, 323, 324
time-share service 18
timing 211
tool 21, 23, 331, 353, 356, 360
budget 21
chaos 182
cost 336
testing 182, 203
training 130, 193, 194, 283, 336
vendor 125, 126, 130, 235, 282
top-down design 19, 214
TQM 214, 215
training 125, 203, 204, 328, 336
cost 331
instruction manuals 165
tuning 359
tutorial 285
undertesting 127
unit 169
bug 94
testing 27, 55, 134, 169, 172, 191, 192, 209,
211, 216, 222-224, 231, 236, 239, 240, 341, 354
university 16
usability 282
testing 92, 197, 198, 281
user 91, 101, 273, 274, 276, 277
dissatisfaction 297
expectations 120, 280
feedback 62
interface 60, 62
involvement 188, 189, 288
life-cycle 61
naive 120
needs 145, 281
profile 11, 12, 59, 159, 196, 281, 286, 295, 296,
299, 348 ,350
questions 62
service 355
-specific testing 359 ,361
support 355
vampire myth 89
verification
criteria 103-105
virtual memory 210
virtual processor interface 146, 147
Volvo 156, 157
walkthrough 214, 215
wish list 145
women 323
engineer 256
workaround 59, 157
worst practice 235
Y2K 171, 223, 237, 238, 311, 346, 356
youth 278, 279
zero-defects 144
Boris Beizer Ph.D. Seminars and Consulting
1232 Glenbrook Road on Software Testing and
Huntingdon Valley, PA 19006 Quality Assurance
TEL: 215-572-5580
FAX: 215-886-0144
Email bbe...@sprintmail.com
------------------------------------------
APPENDIX
BUG STATISTICS AND TAXONOMY
NOTICE--The author gives the reader the right to copy and use this appendix
(and only this appendix) subject to the following restrictions:
1. If you use more than 25% of the taxonomy, the copy must bear, on
the title page:
"Reprinted by written permission of Boris Beizer, from the
second edition of Software Testing Techniques, Van Nostrand
Reinhold, New York, 1990. The author retains all beneficial
rights thereto."
2. If you extract substantial parts (under 25%), the copy must bear,
on the title page:
"Based on Bug Taxonomy and Statistics Appendix, Software
Testing Techniques, Boris Beizer, Van Nostrand Reinhold, New
York, 1990, by written permission of the author."
EXPLANATION AND NOTATION--This document is a taxonomy for bugs. I had
intended at first to adopt the IEEE draft standard on bug classification
(ANSI87E--P1044/D3) but decided not to be because it doesn't cover the
territory.
Bugs are categorized by a 4-digit number, perhaps with subnumbers using
the point system; e.g., "1234.1.6." The "x" which appears is a place
holder for possible future filling in of numbers as the taxonomy is
(inevitably) expanded. For example:
3xxx--structural bugs in the implemented software
32xx--processing bugs
322x--expression evaluation
3222--arithmetic expressions
3222.1--wrong operator
The last digit in a set is always a 9: e.g., 9xxx, 39xx, 329x, 3229,
3226.9. This is the category to use when a finer breakdown is not
available: e.g., an unclassified bug is a 9xxx bug, an unclassified
structural bug is designated as 39xx, an unclassified processing bug is
329x, unclassified expression evaluation bug = 3229, unclassified
arithmetic bug is a 3222.9 bug.
I've adopted this numbering plan and convention so that I can combine
data from various sources which are based on different taxonomies to
different levels of detail.
BUG STATISTICS
This is a spreadsheet dump for combined bug statistics gathered from many
different sources: AKIY71, BELF79, BOEH75A, BOIE72, BONN88, DNIE78, DRAB88,
ELSH76B, ENDR75, EVAN88, GANN76, GILB77, GOEL78B, HAUG64, HOFF77, ITOH73,
LITE76, NASA88, RADA81, REIF79, RUBE75, SCHI78, SCHN75, SCHN79A, SCHW71,
SHOO75, THAY76, and WAGO73. The columns denote:
1. Category Number: 1xxx, 11xx, 119x....
2. Bug Category Name: e.g., requirements incorrect.
3. Bug Type or Subtype Count: e.g., 111x has a count of 222.
4-6. Subgroup Count: e.g., 11xx = 649, 13xx = 224, 15xx = 13.
5-7. Group Count: e.g., 1xxx=1317, 2xxx=2624. Note that type,
subgroup, and group counts progress from left to right. How many
columns are used depends how fine is the group breakdown.
8. Total Percentage. The column shows the percentage of this bug
category compared to the total number of bugs: e.g., 1xxx = 8.12%
of the whole, 11xx = 4.00% and 111x = 1.37%.
9. Group Percentages. The column percentages relative to the group. You
can determine the group by going up to the nearest 100% entry: e.g.,
11xx (incorrect requirements) is 49.28% of all requirements bugs, while
ambiguous requirements are 0.15% of all requirements bugs.
10. Subgroup Percentages. As for group percentages, except for
subgroups: e.g., case completeness (23xx) can be subdivided into
missing (75.13%), duplicated or overlapped (5.18%), extraneous
output (18.65%), and miscellaneous (1.04%).
TOTAL SUM1 SUM2 SUM3 SUM4 SUM5 Percent Percent Percent
KLOC (WITH COMMENTS) 6877.26 of
total Group Subgroup
TOTAL NUMBER OF BUGS 16209 16209 16209 16209 16209 16209
BUGS/KLOC 2.36
1xxx FUNCTIONAL, REQUIREMENTS
1317 1317 1317 8.12% 100.00%
11xx REQUIREMENTS INCORRECT 649 649
4.00% 49.28% 100.00%
111x Incorrect 222
1.37% 16.86% 34.21%
114x Ambiguous 2
0.01% 0.15% 0.31%
119x Other requirements 425
2.62% 32.27% 65.49%
12xx REQUIREMENTS LOGIC 153
0.94% 11.62% 100.00%
124x Inconsistent, incompatible 62
0.38% 4.71% 40.52%
1249 Other inconsistencies 62
0.38% 4.71% 40.52%
129x Other requirements logic problem 91 91
0.56% 6.91% 59.48%
13xx REQUIREMENTS COMPLETENESS 224 224
1.38% 17.01% 100.00%
131x Incomplete 138
0.85% 10.48% 61.61%
132x Missing, not specified 84
0.52% 6.38% 37.50%
139x Other completeness problems 2
0.01% 0.15% 0.89%
15xx PRESENTATION, DOCUMENTATION 13 13
0.08% 0.99% 100.00%
152x Presentation, documentation 1
0.01% 0.08% 7.69%
159x Other presentation problems 12
0.07% 0.91% 92.31%
16xx REQUIREMENTS CHANGES 278
1.72% 21.11% 100.00%
162x Features 175
1.08% 13.29% 62.95%
1621 Feature added 37
0.23% 2.81% 13.31%
1622 Feature deleted 3
0.02% 0.23% 1.08%
1623 Feature changed 110
0.68% 8.35% 39.57%
1629 Other feature changes 25
0.15% 1.90% 8.99%
164x Domain changes 5 5
0.03% 0.38% 1.80%
165x User messages and diagnostics 8 8
0.05% 0.61% 2.88%
167x External interfaces 22 22
0.14% 1.67% 7.91%
169x Other requirements changes 68 68
0.42% 5.16% 24.46%
2xxx FUNCTIONALITY AS IMPLEMENTED 2624 2624 2624
2624 16.19% 100.00%
21xx CORRECTNESS 456
2.81% 17.38% 100.00%
211x Feature not understood 70
0.43% 2.67% 15.35%
218x Feature interaction 32
0.20% 1.22% 7.02%
219x Other feature bugs 354
2.18% 13.49% 77.63%
22xx COMPLETENESS, FEATURES 231
1.43% 8.80% 100.00%
221x Missing feature 56
0.35% 2.13% 24.24%
223x Duplicated, overlapped feature 155
0.96% 5.91% 67.10%
229x Other feature completeness 20
0.12% 0.76% 8.66%
23xx COMPLETENESS, CASES 193
1.19% 7.36% 100.00%
231x Missing case 145
0.89% 5.53% 75.13%
233x Duplicated, overlapped case 10
0.06% 0.38% 5.18%
234x Extraneous Output data 36
0.22% 1.37% 18.65%
239x Other case completeness bugs 2
0.01% 0.08% 1.04%
TOTAL SUM1 SUM2 SUM3 SUM4
SUM5 Percent Percent Percent
Total Group Subgroup
24xx DOMAIN BUGS 778
4.80% 29.65% 100.00%
241x Domain misunderstood, wrong 306
1.89% 11.66% 39.33%
242x Boundary location error 457
2.82% 17.42% 58.74%
243x Boundary closure 11
0.07% 0.42% 1.41%
249x Other domain bugs 4
0.02% 0.15% 0.51%
25xx USER MESSAGES AND DIAGNOSTICS 857 857
5.29% 32.66% 100.00%
26xx EXCEPTION CONDITION MISHANDLED 79 79
0.49% 3.01% 100.00%
29xx OTHER FUNCTIONAL BUGS 30 30
0.19% 1.14% 100.00%
3xxx STRUCTURAL BUGS
4082 25.18% 100.00%
31xx CONTROL FLOW AND SEQUENCING 2078 2078 2078
12.82% 50.91% 100.00%
311x General structure 155
0.96% 3.80% 7.47%
3119 Other general structure 155
0.96% 3.80% 7.47%
312x Control logic and predicates 561
3.46% 13.74% 26.99%
3128 Other control flow predicate bug 268
1.65% 6.56% 12.90%
3129 Other control flow logic bugs 293
1.81% 7.18% 14.10%
314x Loops and iterations 120
0.74% 2.94% 5.77%
3142 Terminal value or condition 54
0.33% 1.32% 2.60%
3144 Iteration variable processing 1
0.01% 0.02% 0.05%
3149 Other loop and iteration 65
0.40% 1.59% 3.13%
315x Control initialization and/or stop 10
0.06% 0.24% 0.48%
3159 Other control state bugs 10
0.06% 0.24% 0.48%
319x Other control flow, sequencing 1232 1232
7.60% 30.18% 59.28%
32xx PROCESSING 2004
12.36% 49.09% 100.00%
321x Algorithmic, fundamental 121 121 121 121
0.75% 2.96% 6.04%
322x Expression evaluation 0 0 445
2.75% 10.90% 22.21%
3222 Arithmetic expressions 278 278
1.72% 6.81% 13.87%
3222.3 Sign 30
0.19% 0.73% 1.50%
3222.9 Other arithmetic 248
1.53% 6.08% 12.38%
3224 Logic, boolean, not control predicate 167 167 167
1.03% 4.09% 8.33%
323x Initialization 303 303 303 303
1.87% 7.42% 15.12%
324x Cleanup 10 10 10 10
0.06% 0.24% 0.50%
325x Precision, accuracy 88 88 88 88
0.54% 2.16% 4.39%
326x Execution time 47 47 47 47
0.29% 1.15% 2.35%
329x Other processing 990 990 990 990
6.11% 24.25% 49.40%
4xxx DATA 3638 3638
3638 22.44% 100.00%
41xx DATA DEFINITION, STRUCTURE, DECLARATION 1805
11.14% 49.62% 100.00%
413x Static initial or default value 157
0.97% 4.32% 8.70%
414x Duplication and aliases 11
0.07% 0.30% 0.61%
4149 Other duplication and aliases 11
0.07% 0.30% 0.61%
419x Other data structure definition 1637 1637
10.10% 45.00% 90.69%
42xx DATA ACCESS AND HANDLING 1831
11.30% 50.33% 100.00%
421x Type 359
2.21% 9.87% 19.61%
4212 Wrong type 10
0.06% 0.27% 0.55%
4214 Type transformation 84
0.52% 2.31% 4.59%
4216 Scaling, units 237
1.46% 6.51% 12.94%
4219 Other type bugs 28
0.17% 0.77% 1.53%
TOTAL SUM1 SUM2 SUM3 SUM4
SUM5 Percent Percent Percent
Total Group Subgroup
423x Value 236
1.46% 6.49% 12.89%
4232 Initialization 236
1.46% 6.49% 12.89%
424x Duplication and aliases 10
0.06% 0.27% 0.55%
4249 Other duplication and aliases 10
0.06% 0.27% 0.55%
426x Resources 11
0.07% 0.30% 0.60%
4269 Other dynamic resource 11
0.07% 0.30% 0.60%
428x Access 677
4.18% 18.61% 36.97%
4281 Wrong object accessed 244
1.51% 6.71% 13.33%
4282 Access rights violation 8
0.05% 0.22% 0.44%
4283 Data flow anomaly 115
0.71% 3.16% 6.28%
4285 Saving or protecting bug 10
0.06% 0.27% 0.55%
4287 Access mode, direct/indirect 113
0.70% 3.11% 6.17%
4288 Object boundary, structure 115
0.71% 3.16% 6.28%
4289 Other access bug 72
0.44% 1.98% 3.93%
429x Other data access and handling 538 538
3.32% 14.79% 29.38%
49xx OTHER DATA PROBLEMS 2 2 2
0.01% 0.05% 100.00%
5xxx IMPLEMENTATION 1601 1601
1601 9.88% 100.00%
51xx CODING AND TYPOGRAPHICAL 322 322
1.99% 20.11% 100.00%
511x Coding wild card, typographical 26
0.16% 1.62% 8.07%
519x Other, general coding bugs 296
1.83% 18.49% 91.93%
52xx STANDARDS VIOLATION 318
1.96% 19.86% 100.00%
527x Format 15 15
0.09% 0.94% 4.72%
528x Comments 68 68
0.42% 4.25% 21.38%
529x Other standards, style violation 235 235
1.45% 14.68% 73.90%
53xx DOCUMENTATION 960 960
5.92% 59.96% 100.00%
531x Incorrect 550
3.39% 34.35% 57.29%
532x Inconsistent 9
0.06% 0.56% 0.94%
534x Incomplete 146
0.90% 9.12% 15.21%
539x Other documentation, general 255
1.57% 15.93% 26.56%
59xx OTHER IMPLEMENTATION 1 1 1
0.01% 0.06% 100.00%
6xxx INTEGRATION 1455 1455
1455 8.98% 100.00%
61xx INTERNAL INTERFACES 859
5.30% 59.04% 100.00%
611x Component invocation 27
0.17% 1.86% 3.14%
6119 Other component invocation 27
0.17% 1.86% 3.14%
612x Interface parameter, invocation 128
0.79% 8.80% 14.90%
6121 Wrong parameter 75
0.46% 5.15% 8.73%
6126 Parameter sequence 2
0.01% 0.14% 0.23%
6129 Other invocation parameter bugs 51
0.31% 3.51% 5.94%
613x Component return 70
0.43% 4.81% 8.15%
6131 Parameter identity wrong 37
0.23% 2.54% 4.31%
6139 Other parameter bugs on return 33
0.20% 2.27% 3.84%
614x Initialization, state 221 221
1.36% 15.19% 25.73%
619x Other, internal interfaces 413 413
2.55% 28.38% 48.08%
TOTAL SUM1 SUM2 SUM3 SUM4
SUM5 Percent Percent Percent
Total Group Subgroup
62xx EXTERNAL INTERFACES AND TIMING 518
3.20% 35.60% 100.00%
621x Interrupts 94 94
0.58% 6.46% 18.15%
622x Devices, drivers 137
0.85% 9.42% 26.45%
6222 Device, driver, init and/or state 3
0.02% 0.21% 0.58%
6224 Device, driver, command bug 28
0.17% 1.92% 5.41%
6226 Return data, status, misinterpreted 24
0.15% 1.65% 4.63%
6229 Other device, driver 82
0.51% 5.64% 15.83%
623x I/O timing and throughput 23 23
0.14% 1.58% 4.44%
629x Other, external interface, timing 264 264
1.63% 18.14% 50.97%
69xx OTHER INTEGRATION 78 78 78
0.48% 5.36% 100.00%
7xxx SYSTEM, SOFTWARE ARCHITECTURE 282 282 282
282 1.74% 100.00%
71xx O/S CALL, USE BUG 47
0.29% 16.67% 100.00%
711x Invocation, command 5
0.03% 1.77% 10.64%
714x Space 3
0.02% 1.06% 6.38%
719x Other OS Call, use bugs 39
0.24% 13.83% 82.98%
72xx SOFTWARE ARCHITECTURE 139
0.86% 49.29% 100.00%
721x Interlocks and semaphores 56
0.35% 19.86% 40.29%
724x Resource Management and Control 8
0.05% 2.84% 5.76%
729x General software architecture 75
0.46% 26.60% 53.96%
73xx RECOVERY AND ACCOUNTABILITY 4 4
0.02% 1.42% 100.00%
74xx PERFORMANCE 64
0.39% 22.70% 100.00%
742x Response time, delay 44
0.27% 15.60% 68.75%
749x Other performance, unspecified 20
0.12% 7.09% 31.25%
75xx INCORRECT DIAGNOSTIC, EXCEPTIONS 16 16
0.10% 5.67% 100.00%
76xx PARTITIONS, OVERLAYS 3 3
0.02% 1.06% 100.00%
77xx SYSGEN/ENVIRONMENT 9 9
0.06% 3.19% 100.00%
8xxx TEST DEFINITION OR EXECUTION 447 447 447
447 2.76% 100.00%
81xx TEST DESIGN BUGS 11
0.07% 2.46% 100.00%
811x Requirements misunderstood 3
0.02% 0.67% 27.27%
819x Other test design bugs 8
0.05% 1.79% 72.73%
82xx TEST EXECUTION BUGS 355
2.19% 79.42% 100.00%
823x Data base 6
0.04% 1.34% 1.69%
824x Configuration 66
0.41% 14.77% 18.59%
828x Verification act 28
0.17% 6.26% 7.89%
829x Other test execution bugs 255
1.57% 57.05% 71.83%
83xx TEST DOCUMENTATION 11 11
0.07% 2.46% 100.00%
84xx TEST CASE COMPLETENESS 64 64
0.39% 14.32% 100.00%
89xx OTHER TEST DESIGN/EXECUTION BUGS 6 6
0.04% 1.34% 100.00%
9xxx OTHER BUGS, UNSPECIFIED 763 763 763 763 763
763 4.71% 100.00% 100.00%
BUG TAXONOMY
1xxx: FUNCTIONAL BUGS: REQUIREMENTS AND FEATURES: bugs having to do
with requirements as specified or as implemented.
11xx: REQUIREMENTS INCORRECT: the requirement or a part of it is
incorrect.
111x: Incorrect: requirement is wrong.
112x: Undesirable: requirement is correct as stated but it is not
desirable.
113x: Not needed: requirement is not needed.
12xx: LOGIC: the requirement is illogical or unreasonable.
121x: Illogical: illogical, usually because of a
self-contradiction which can be exposed by a logical
analysis of cases.
122x: Unreasonable: logical and consistent but unreasonable with
respect to the environment and/or budgetary and time
constraints.
123x: Unachievable: requirement fundamentally impossible or cannot be
achieved under existing constraints.
124x: Inconsistent, incompatible: requirement is inconsistent with
other requirements or with environment.
1242: Internal: the inconsistency is evident within the
specified component.
1244: External: the inconsistency is with external (to the
component) components or the environment.
1248: Configuration sensitivity: the incompatibility is with one or
more configurations (hardware, software, operating system) in
which the component is expected to work.
13xx: COMPLETENESS: the requirement as specified is either
ambiguous, incomplete, or overly specified.
131x: Incomplete: the specification is incomplete; cases,
features, variations or attributes are not specified and
therefore not implemented.
132x: Missing, unspecified: the entire requirement is missing.
133x: Duplicated, overlapped: specified requirement totally or
partially overlaps another requirement either already
implemented or specified elsewhere.
134x: Overly generalized: requirement as specified is correct and
consistent but is overly generalized (e.g., too powerful) for the
application.
137x: Not downward compatible: requirement as specified will mean that
objects created or manipulated by prior versions can either not
be processed by this version or will be incorrectly processed.
138x: Insufficiently extendable: requirement as specified cannot
be expanded in ways which are likely to be needed--important
hooks left out of specification.
14xx: VERIFIABILITY: specification bugs having to do with verifying that
the requirement was correctly or incorrectly implemented.
141x: Unverifiable: the requirement, if implemented, cannot be
verified by any means or within available time and budget.
For example, it is possible to design a test but the outcome
of the test cannot be verified as correct or incorrect.
142x: Untestable: it is not possible to design and/or execute
tests which will verify the requirement--untestable is
stronger than unverifiable.
15xx: PRESENTATION: bugs in the presentation or documentation of
requirements. The requirements are presumed to be correct,
but the form in which they are presented is not. This can be
important for test design automation systems which demand
specific formats.
152x: Presentation, documentation: general presentation,
documentation, format, media, etc.
153x: Standards: presentation violates standards for requirements.
16xx: REQUIREMENT CHANGES: requirements, whether or not correct, have
been changed between the time programming started and testing
ended.
162x: Features: requirement changes concerned with features.
1621: feature added: a new feature has been added.
1622: feature deleted: previously required feature deleted.
1623: feature changed: significant changes to feature, other
than changes in cases.
163x: Cases: cases within a feature have been changed. Feature
itself is not significantly modified except for cases.
1631: cases added.
1632: cases deleted.
1633: cases changed: processing or treatment of specific case(s)
changed.
164x: Domain changes: input data domain modified: e.g., boundary
changes, closure, treatment.
165x: User messages and diagnostics: changes in text, content, or
conditions under which user prompts, warning, error messages,
etc. are produced.
166x: Internal interfaces: direct internal interfaces such as call
sequences, or indirect interfaces (e.g., via data structures)
have been changed.
167x: External interfaces: external interfaces, such as device
drivers, protocols, etc. have been changed.
168x: Performance and timing: changes to performance requirements
(e.g., throughput) and/or timings.
2xxx: FUNCTIONALITY AS IMPLEMENTED: requirement known or assumed to be
correct, implementable, and testable, but implementation is
wrong.
21xx: CORRECTNESS: having to do with the correctness of the
implementation.
211x: Feature misunderstood, wrong: feature as implemented is not
correct--not as specified.
218x: Feature interactions: feature is correctly implemented by
itself, but has incorrect interactions with other features,
or specified or implied interaction is incorrectly handled.
22xx: COMPLETENESS, FEATURES: having to do with the completeness
with which features are implemented.
221x: Missing feature: an entire feature is missing.
222x: Unspecified feature: a feature not specified has been
implemented.
223x: Duplicated, overlapped feature: feature as implemented
duplicates or overlaps features implemented by other parts
of the software.
23xx: COMPLETENESS, CASES: having to do with the completeness of
cases within features.
231x: Missing case.
232x: Extra case: cases which should not have been handled are.
233x: Duplicated, overlapped case: duplicated handling of cases or
partial overlap with other cases.
234x: Extraneous output data: data not required is output.
24xx: DOMAINS: processing case or feature depends on a combination of
input values. A domain bug exists if the wrong processing is
executed for the selected input value combination.
241x: Domain misunderstood, wrong: misunderstanding of the size,
shape, boundaries, or other characteristics of the specified
input domain for the the feature or case. Most bugs related
to handling extreme cases are domain bugs.
242x: Boundary locations: the values or expressions which define a
domain boundary are wrong: e.g., "X>=6" instead of "X>=3."
243x: Boundary closures: end points and boundaries of the domain
are incorrectly associated with an adjacent domain: e.g.,
"X>=0" instead of "X>0."
244x: Boundary intersections: domain boundaries are defined by a
relation between domain control variables. That relation,
as implemented, is incorrect: e.g., "IF X>0 AND Y>0
..." instead of "IF X>0 OR Y>0..."
25xx: USER MESSAGES AND DIAGNOSTICS: user prompt or printout or
other form of communication is incorrect. Processing is
assumed to be correct: e.g., a false warning, failure to
warn, wrong message, spelling, formats.
26xx: EXCEPTION CONDITIONS MISHANDLED: exception conditions such as
illogicals, resource problems, failure modes, which require
special handling are not correctly handled or by the wrong
exception handling mechanisms.
3xxx: STRUCTURAL BUGS: bugs related to the component's structure:i.e.,
the code.
31xx: CONTROL FLOW AND SEQUENCING: bugs specifically related to the
control flow of the program or the order and extent to which
things are done, as distinct from what is done.
311x: General structure: general bugs related to component structure.
3112: Unachievable path: a functionally meaningful processing path in
the code for which there is no combination of input values
which will force that path to be executed. Do not confuse with
unreachable code. The code in question might be reached by
some other path.
3114: Unreachable code: code for which there is no combination
of input values which will cause that code to be executed.
3116: Dead end code: code segments which once entered cannot be
exited, even though it was intended that an exit be possible.
312x: Control logic and predicates: the path taken through a
program is directed by control flow predicates (e.g.,
boolean expressions). This category addresses the
implementation of such predicates.
3122: Duplicated logic: control logic which should appear only once
is inadvertently duplicated in whole or in part.
3124: Don't care: improper handling of cases for which what is to be
done does not matter either because the case is impossible or
because it really doesn't matter: e.g., incorrectly assuming
that the case is a don't care case, failure to do case
validation, not invoking the correct exception handler,
improper logic simplification to take advantage of such cases.
3126: Illogicals: improper identification of, or processing of
illogical or impossible conditions. An illogical is stronger
than a don't care. Illogicals usually mean that something bad
has happened and that recovery is needed. Examples of bugs
include: illogical not really so, failure to recognize
illogical, invoking wrong handler, improper simplification of
control logic to take advantage of the case.
3128: Other control flow predicate bugs: control flow problems which
can be directly attributed to the incorrect formulation of a
control flow predicate: e.g., "IF A>B THEN..." instead of
"IF A<B THEN..."
313x: Case selection bug: simple bugs in case selections, such as
improperly formulated case selection expression, GOTO list,
or bug in assigned GOTO.
314x: Loops and iteration: bugs having to do with the control of
loops.
3141: Initial value: initial iteration value wrong: e.g.,
"FOR I 3 TO 17..." instead of "FOR I = 8 TO 17."
3142: Terminal value or condition: value, variable, or
expression used to control loop termination is incorrect:
e.g., "FOR I = 1 TO 7..." instead of "FOR I = 1 TO 8."
3143: Increment value: value, variable, or expression used to
control loop increment value is incorrect: e.g., "FOR I =
1 TO 7 STEP 2..." instead of "FOR I = 1 TO 7 STEP 5.."
3144: Iteration variable processing: where end points and/or
increments are controlled by values calculated within the
loop's scope, a bug in such calculations.
3148: Exception exit condition: where specified values
or conditions or relations between variables force an
abnormal exit to the loop, either incorrect processing of
such conditions or incorrect exit mechanism invoked.
315x: Control initialization and/or state: bugs having to do with
how the program's control flow is initialized and changes of
state which affect the control flow: e.g., switches.
3152: Control initialization: initializing to the wrong state or
failing to initialize.
3154: Control state: for state-determined control flows, incorrect
transition to a new state from the current state: e.g., input
condition X requires a transition to state B, given that the
program is in state A, instead, the transition is to state C.
Most incorrect GOTO's are included in this category.
316x: Incorrect exception handling: any incorrect invocation of a
control flow exception handler not previously categorized.
32xx: PROCESSING: bugs related to processing under the assumption
that the control flow is correct.
321x: Algorithmic, fundamental: inappropriate or incorrect algorithm
selected, but implemented correctly: e.g., using an incorrect
approximation, using a shortcut string search algorithm that
assumes string characteristics which may not apply.
322x: Expression evaluation: bugs having to do with the way arithmetic,
boolean, string, and other expressions are evaluated.
3222: Arithmetic: bugs related to evaluation of arithmetic
expressions.
3222.1: Operator: wrong arithmetic operator or function used.
3222.2: Parentheses: syntactically correct bug in placement of
parentheses or other arithmetic delimiters.
3222.3: Sign: bug in use of sign.
3224: Logical or boolean, not control: bug in the manipulation or
evaluation of boolean expressions which are not (directly) part
of control flow predicates: e.g., using wrong mask, AND instead
of OR, incorrect simplification of boolean function.
3226: String manipulation: bug in string manipulation.
3226.1: Beheading: the beginning of a string is cut off when it
should not have been or not cut off when it should be.
3226.2: Curtailing: as for beheading but for string end.
3226.3: Concatenation order: strings are concatenated in wrong
order or concatenated when they should not be.
3226.3.1: Append instead of precede.
3226.3.2: Precede instead of append.
3226.4: Inserting: having to do with the insertion of one
string into another.
3226.5: Converting case: case conversion (upper to lower, say)
is incorrect.
3226.6: Code conversion: string is converted to another code
incorrectly or not converted when it should be.
3226.7: Packing, unpacking: strings are incorrectly packed or
unpacked.
3228: Symbolic, algebraic: bugs in symbolic processing of
algebraic expressions.
323x: Initialization: bugs in initialization of variables,
expressions, functions, etc. used in processing, excluding
initialization bugs associated with declarations and data
statements and loop initialization.
324x: Cleanup: incorrect handling of cleanup of temporary data
areas, registers, states, etc. associated with processing.
325x: Precision, accuracy: insufficient or excessive precision,
insufficient accuracy and other bugs related to
number representation system used.
326x: Execution time: excessive (usually) execution time for
processing component.
4xxx: DATA: bugs in the definition, structure, or use of data.
41xx: DATA DEFINITION, STRUCTURE, DECLARATION: bugs in the
definition, structure and initialization of data: e.g., in
DATA statements. This category applies whether the object is
declared statically in source code or created dynamically.
411x: Type: the data object type, as declared, is incorrect: e.g.,
integer instead of floating, short instead of long, pointer
instead of integer, array instead of scalar, incorrect user
defined type.
412x: Dimension: for arrays and other objects which have a dimension
(e.g., arrays, records, files) by which component objects can be
indexed, a bug in the dimension or in the minimum or maximum
dimensions, or in redimensioning statements.
413x: Initial, default values: bugs in the assigned initial values of
the object (e.g., in DATA statements), selection of incorrect
default values, or failure to supply a default value if needed.
414x: Duplication and aliases: bugs related to the incorrect
duplication or failure to create a duplicated object.
4142: Duplicated: duplicated definition of an object where
allowed by the syntax.
4144: Aliases: object is known by one or more aliases but specified
alias is incorrect; object not aliased when it should have
been.
415x: Scope: the scope, partition, or components to which the
object applies is incorrectly specified.
4152: Local should be global: a locally defined object (e.g., within
the scope of a specific component) should have been specified
more globally (e.g., in COMMON).
4154: Global should be local: the scope of an object is too
global, it should have been declared more locally.
4156: Global/local inconsistency or conflict: a syntactically
acceptable conflict between a local and/or global
declaration of an object (e.g., incorrect COMMON).
416x: Static/dynamic resources: related to the declaration of
static and dynamically allocated resources.
4162: Should be static resource: resource is defined as a
dynamically allocated object but should have been static
(e.g., permanent).
4164: Should be dynamic resource: resource is defined as static
but should have declared as dynamic.
4166: Insufficient resources, space: number of specified resources is
insufficient or there is insufficient space (e.g., main memory,
cache, registers, disc, etc.) to hold the declared resources.
4168: Data overlay bug: data objects are to be overlayed but
there is an bug in the specification of the overlay areas.
42xx: DATA ACCESS AND HANDLING: having to do with access and manipulation
of data objects which are presumed to be correctly defined.
421x: Type: bugs having to do with the object type.
4212: Wrong type: object type is incorrect for required
processing: e.g., multiplying two strings.
4214: Type transformation: object undergoes incorrect type
transformation: e.g., integer to floating, pointer to integer),
specified type transformation is not allowed, required type
transformation not done. Note, type transformation bugs can
exist in any language, whether or not it is strongly typed,
whether or not there are user defined types.
4216: Scaling, units: scaling or units (semantic) associated with
object is incorrect, incorrectly transformed, or not
transformed: e.g., FOOT-POUNDS to STONE-FURLONGS.
422x: Dimension: for dynamically variable dimensions of a
dimensioned object, a bug in the dimension: e.g., dynamic
redimension of arrays, exceeding maximum file length,
removing one more than the minimum number of records.
423x: Value: having to do with the value of data objects or parts
thereof.
4232: Initialization: initialization or default value of object is
incorrect. Not to be confused with initialization and default
bugs in declarations. This is a dynamic initialization bug.
4234: Constant value: incorrect constant value for an object:
e.g., a constant in an expression.
424x: Duplication and aliases: bugs in dynamic (run time)
duplication and aliasing of objects.
4242: Object already exists: attempt to create an object which
already exists.
4244: No such object: attempted reference to an object which
does not exist.
426x: Resources: having to do with dynamically allocated resources
and resource pools, in whatever memory media they exist:
main, cache, disc, bulk RAM. Included are: queue blocks,
control blocks, buffer blocks, heaps, files, etc.
4262: No such resource: referenced resource does not exist.
4264: Wrong resource type: wrong resource type referenced.
428x: Access: having to do with the access of objects as distinct from
the manipulation of objects. In this context, accesses include
read, write, modify, and (in some instances) create and destroy.
4281: Wrong object accessed: incorrect object accessed: e.g., "X
:= ABC33" instead of "X := ABD33."
4282: Access rights violation: access rights are controlled by
attributes associated with the caller and the object. For
example, some callers can only read the object, others can read
and modify, etc. Violations of object access rights are
included in this category whether or not a formal access rights
mechanism exists: that is, access rights could be specified by
programming conventions rather than by software.
4283: Data flow anomaly: data flow anomalies involve the sequence of
accesses to an object: e.g., reading or initializing an object
before it has been created, creating and then not using.
4284: Interlock bug: where objects are in simultaneous use by more
than one caller, interlocks and synchronization mechanisms may
be used to assure that all data are current and changed by only
one caller at a time. These are not bugs in the interlock or
synchronization mechanism but in the use of that mechanism.
4285: Saving or protecting bug: application requires that the object
be saved or otherwise protected at different program states, or
alternatively, not protected. These are bugs related to the
incorrect usage of such protection mechanisms or procedures.
4286: Restoration bug: application requires that a previously saved
object be restored prior to processing: e.g., POP the stack,
restore registers after interrupt. This category includes bugs
in the incorrect restoration of data objects and not bugs in
the implementation of the restoration mechanism.
4287: Access mode, direct/indirect: object is accessed by wrong
means: e.g., direct access of an object for which indirect
access is required, call by value instead of name or
vice-versa, indexed instead of sequential or vice-versa.
4288: Object boundary or structure: access to object is partly
correct, but the object structure and its boundaries are
handled incorrectly: e.g., fetching 8 characters of a string
instead of 7, mishandling word boundaries, getting too much or
too little of an object.
5xxx: IMPLEMENTATION: bugs having to do with the implementation of the
software. Some of these, such as standards and documentation, may
not affect the actual workings of the software. They are included in
the bug taxonomy because of their impact on maintenance.
51xx: CODING AND TYPOGRAPHICAL: bugs which can be clearly attributed to
simple coding and typographical bugs. Classification of a bug into
this category is subjective. If a programmer believed that the
correct variable, say, was "ABCD" instead of "ABCE," then it would
be classified as a 4281 bug (wrong object accessed). Conversely,
if E was changed to D because of a typewriting bug, then it belongs
here.
511x: Coding wild card, typographical: all bugs which can be
reasonably attributed to typing and other typographical bugs.
512x: Instruction, construct misunderstood: all bugs which can be
reasonably attributed to a misunderstanding of an
instruction's operation or HOL statement's action.
52xx: STANDARDS VIOLATION: bugs having to do with violating or
misunderstanding the applicable programming standards and
conventions. The software is assumed to work properly.
521x: Structure violations: violations concerning control flow
structure, organization of the software, etc.
5212: Control flow: violations of control flow structure
conventions: e.g., excessive IF-THEN-ELSE nesting, not using
CASE statements where required, not following dictated
processing order, jumping into or out of loops, jumping into or
out of decisions.
5214: Complexity: violation of maximum (usually) or minimum (rare)
complexity guidelines as measured by some specified complexity
metric: e.g., too many lines of code in module, cyclomatic
complexity greater than 200, excessive Halstead length, too
many tokens.
5215: Loop nesting depth: excessive loop nesting depth.
5216: Modularity and partition: modularity and partition rules
not followed: e.g., minimum and maximum size, object
scope, functionally dictated partitions.
5217: Call nesting depth: violations of component (e.g.,
subroutine, subprogram, function) maximum nesting depth,
or insufficient depth where dictated.
522x: Data definition, declarations: the form and/or location of
data object declaration is not according to standards.
523x: Data access: violations of conventions governing how data objects
of different kinds are to be accessed, wrong kind of object used:
e.g., not using field access macros, direct access instead of
indirect, absolute reference instead of symbolic, access via
register, etc.
524x: Calling and invoking: bug in the manner in which other
processing components are called, invoked, or communicated
with: e.g., direct subroutine call which should be indirect,
violation of call and return sequence conventions.
526x: Mnemonics, label conventions: violations of the rules by
which names are assigned to objects: e.g., program labels,
subroutine and program names, data object names, file names.
527x: Format: violations of conventions governing the overall format
and appearance of the source code: indentation rules, pagination,
headers, ID block, special markers.
528x: Comments: violations of conventions governing the use, placement,
density, and format of comments. The content of comments in
covered by 53xx, documentation.
53xx: DOCUMENTATION: bugs in the documentation associated with the
code or the content of comments contained in the code.
531x: Incorrect: documentation statement is wrong.
532x: Inconsistent: documentation statement is inconsistent with
itself or with other statements.
533x: Incomprehensible: documentation cannot be understood by a
qualified reader.
534x: Incomplete: documentation correct but important facts missing.
535x: Missing: major parts of documentation missing.
6xxx: INTEGRATION: bugs having to do with the integration of, and
interfaces between, components. The components themselves are
assumed to be correct.
61xx: INTERNAL INTERFACES: bugs related to the interfaces between
communicating components with the program under test. The
components are assumed to have passed their component level tests.
In this context, direct or indirect transfer of data or control
information via a memory object such as tables, dynamically
allocated resources, or files, constitute an internal interface.
611x: Component invocation: bugs having to do with how software
components are invoked. In this sense, a "component" can be a
subroutine, function, macro, program, program segment, or any
other sensible processing component. Note the use of "invoke"
rather than "call," because there may be no actual call as such:
e.g., a task order placed on a processing queue is an invocation
in our sense, though (typically) not a call.
6111: No such component: invoked component does not exist.
6112: Wrong component: incorrect component invoked.
612x: Interface parameter, invocation: having to do with the parameters
of the invocation, their number, order, type, location, values,
etc.
6121: Wrong parameter: parameters of the invocation are
incorrectly specified.
6122: Parameter type: incorrect invocation parameter type used.
6124: Parameter structure: structural details of parameter used
are incorrect: e.g., size, number of fields, subtypes.
6125: Parameter value: value (numerical, boolean, string) of the
parameter is wrong.
6126: Parameter sequence: parameters of the invocation sequence
in the wrong order, too many parameters, too few
parameters.
613x: Component invocation return: having to do with the interpretation
of parameters provided by the invoked component on return to the
invoking component or on release of control to some other
component. In this context, a record, a subroutine return
sequence, or a file can qualify for this category of bug. Note
that the bugs included here are not bugs in the component that
created the return data but in the receiving component's
subsequent manipulation and interpretation of that data.
6131: Parameter identity: wrong return parameter accessed.
6132: Parameter type: wrong return parameter type used. That
is, the component using the return data interprets a
return parameter incorrectly as to type.
6134: Parameter structure: return parameter structure
misinterpreted.
6136: Return Sequence: sequence assumed for return parameters is
incorrect.
614x: Initialization, state: invoked component not initialized or
initialized to the wrong state or with incorrect data.
615x: Invocation in wrong place: the place or state in the invoking
component at which the invoked component was invoked is wrong.
616x: Duplicate or spurious invocation: component should not have
been invoked or has been invoked more often than necessary.
62xx: EXTERNAL INTERFACES AND TIMING: having to do with external
interfaces, such as I/O devices and/or drivers, or other
software not operating under the same control structure. Data
passage by files or messages qualify for this bug category.
621x: Interrupts: bugs related to incorrect interrupt handling or
setting up for interrupts: e.g., wrong handler invoked,
failure to block or unblock interrupts.
622x: Devices and drivers: incorrect interface with devices or device
drivers or incorrect interpretation of return status data.
6222: Device, driver, initialization or state: incorrect
initialization of device or driver, failure to initialize,
setting device to the wrong state.
6224: Device, driver, command bug: bug in the command issued to
a device or driver.
6226: Device, driver, return/status misinterpretation: return status
data from device or driver misinterpreted or ignored.
623x: I/O timing or throughput: bugs having to do with timings and
data rates for external devices such as: not meeting specified
timing requirements (too long or too short), forcing too much
throughput, not accepting incoming data rates.
7xxx: SYSTEM AND SOFTWARE ARCHITECTURE: bugs not attributable to a
component or to the interface between components but which affect the
entire software system or stem from architectural errors in the
system.
71xx: OS bug: bugs related to the use of operating system facilities.
Not to be confused with bugs in the operating system itself.
711x: Invocation, command: erroneous command given to operating
system or OS facility incorrectly invoked.
712x: Return data, status misinterpretation: data returned from
operating system or status information ignored or
misinterpreted.
714x: Space: required memory (cache, disc, RAM) resource not
available or requested in wrong way.
72xx: Software architecture: architectural problems not elsewhere
defined.
721x: Interlocks and semaphores: bugs in the use of interlock
mechanisms and interprocess communications facilities. Not
to be confused with bugs in these mechanisms themselves:
e.g., failure to lock, failure to unlock, failure to set or
reset semaphore, duplicate locking.
722x: Priority: bugs related to task priority: e.g., priority too low
or too high, priority selected not allowed, priority conflicts.
723x: Transaction flow control: where the path taken by a transaction
through the system is controlled by an implicit or explicit
transaction flow control mechanism, these are bugs related to the
definition of such flows. Note that all components and their
interfaces could be correct but this class of bug could still
exist.
724x: Resource management and control: bugs related to the
management of dynamically allocated shared resource objects:
e.g., not returning a buffer block after use, not getting an
object, failure to clean up an object after use, getting
wrong kind of object, returning object to wrong pool.
725x: Recursive calls: bugs in the use of recursive invocation of
software components or incorrect recursive invocation.
726x: Reentrance: bugs related to reentrance of program
components: e.g., a reentrant component which should not be
reentrant, nonreentrant component which should be, reentrant
call which should be nonreentrant.
73xx: RECOVERY, ACCOUNTABILITY: bugs related to the recovery of objects
after failure and to the accountability for objects despite
failures.
74xx: PERFORMANCE: bugs related to the throughput-delay behavior of
software under the assumption that all other aspects are correct.
741x: Throughput inadequate.
742x: Response time, delay: response time to incoming events too long
at specified load or too short (rare), delay between outgoing
events too long or too short.
743x: Insufficient users: maximum specified number of simultaneous
users or task cannot be accommodated at specified
transaction delays.
748x: Performance parasites: any bug whose primary or only symptom
is a performance degradation: e.g., the harmless but
needless repetition of operations, fetching and returning
more dynamic resources than needed.
75xx: INCORRECT DIAGNOSTIC, EXCEPTION: diagnostic or error message
incorrect or misleading. Exception handler invoked is wrong.
76xx: PARTITIONS AND OVERLAYS: memory or virtual memory is
incorrectly partitioned, overlay to wrong area, overlay or
partition conflicts.
77xx: SYSGEN OR ENVIRONMENT: wrong operating system version, incorrect
system generation, or other host environment problem.
8xxx: TEST DEFINITION OR EXECUTION BUGS: bugs in the definition,
design, execution of tests or the data used in tests. These are
as important as "real" bugs.
81xx: DESIGN BUGS: bugs in the design of tests.
811x: Requirements misunderstood: test and component are mismatched
because test designer did not understand requirements.
812x: Incorrect outcome predicted: predicted outcome of test does
not match required or actual outcome.
813x: Incorrect path predicted: outcome is correct but achieved by
wrong predicted path. The test is only coincidentally correct.
814x: Test initialization: specified initial conditions for test
are wrong.
815x: Test data structure or value: data objects used in tests or
their values are wrong.
816x: Sequencing bug: the sequence in which tests are to be executed,
relative to other tests or to test initialization, is wrong.
817x: Configuration: the hardware and/or software configuration
and/or environment specified for the test is wrong.
818x: Verification method, criteria: the method by which the
outcome will be verified is incorrect or impossible.
82xx: EXECUTION BUGS: bugs in the execution of tests as contrasted
with bugs in their design.
821x: Initialization: tested component not initialized to the
right state or values.
822x: Keystroke or command: simple keystroke or button hit error.
823x: Data base: data base used to support the test wrong.
824x: Configuration: configuration and/or environment specified
for the test was not used during the run.
828x: Verification act: the act of verifying the outcome was
incorrectly executed.
83xx: TEST DOCUMENTATION: documentation of test case or verification
criteria is incorrect or misleading.
84xx: TEST CASE COMPLETENESS: cases required to achieve specified
coverage criteria missing.