Usability isn't testing
boris beizer
Let me start this essay by quenching the inevitable flames, hopefully before
they flare too brightly. I do believe that there are usability issues in
software and that usability design is important for some parts of some
software. I believe that that part of that software that has an interface
with humans should be designed with the human being in mind. I also believe
that there is a professional discipline involved in assessing how usable a
piece of software is and furthermore, that there are design principles
stemming from that discipline. So if you're going to flame me with a long
diatribe on how important usability is -- don't bother because I agree with
you. If you're going to cite an extensive social-science literature on
usability assessment and a long history of that discipline -- don't bother
because I know about and acknowledge that literature and expertise.
The point of this essay is that the two disciplines, software testing and
usability assessment (note that I do not use the term "usability testing" for
reasons that will be made clear in the sequel), have very little in common
and that both disciplines are damaged and frustrated in their development as a
result of a largely accidental conjunction under the term "software testing."
2. A Holistic Approach.
Popular term "holistic." We should take a holistic approach to testing --
certainly as a philosophical underpinning. To me, that has always implied
balance. Not only should we recognize that the whole can be more than the sum
of its parts, but that those parts play a proportional role. Not all parts
are equally important. Engineering is all about optimization of conflicting
requirements in the light of limited resources. That means that effort
allocated to developing a product's component should be proportioned to the
importance of that component. We do not want the tail to wag the dog, never
mind the fringe hairs on the tip of that tail wagging the dog.
So let's set a perspective on the relative importance of usability issues in
the entire software development world and not just that tail-tip that
interfaces with humans. Let's start with software that does have a human
interface. I estimate that at most 15% of a software product that does have a
human interface (e.g., PC application) has any usability concern at all. In
fact, at most 15% of that software has to do with the application at all --
the rest being infrastructure without which the application can't be built.
Furthermore, the increased use of components to implement major parts of
applications means that things that obviously interface with people are done
by packaged components -- menu generators, help generators, installation
generators, to name a few. For such implementations there are no software
testing issues at all because most of the human interface issues are embedded
in data. And by changing that data one can go from a horrible human
interface to a superb one, or vice-versa, without changing the software and
therefore, without any software testing impact whatsoever. We see this
already in the contemporary approach to localization. There are localization
specialists of course and they are important if you intend to sell a PC
product in some language other than English or in some location outside of the
US. But does localization concerns, as important as they are for some
software, have anything to do with software testing? No. It is a totally a
design issue that can be determined long before any software is written.
Furthermore, people concerned with localization do their work on a base of
well-tested software that probably has an English/US implementation.
<footnote 1.> So our first point in taking a holistic point of view is to
recognize that in software that does interfaces with humans that interface is
confined to a small part of the whole and technological advances are rapidly
taking such concerns totally out of software (and therefore, out of software
testing) and putting it in data.
--------------------------
<footnote 1> I know that Japanese and Chinese require a different basis for
major parts because of a 16 bit character set as contrasted with an 8 bit set.
And obviously, Arabic, Hebrew, and Amharic have similar localization issues
that get a bit deeper into the infrastructure than merely filling out a German
table as contrasted with an English table.
--------------------------
Another example of what I'm talking about is your familiar ATM systems.
That has an obvious human interface. But how much of the involved software
interfaces with humans in any significant way? Very little and much of the
part that does is badly done and furthermore has nothing to do with software.
<footnote 2>. You, as a user, can initiate at most 1/3 of the transactions
involved in an ATM. One third has to do with test and diagnostics to make
sure the hardware is working correctly, and another third has to do with
bookkeeping and auditing to make sure no employee steals anything (and
auditors are hardly human, after all). Looking then just at your side of the
ATM, most of the transactions are transparent to you, albeit very complicated.
There's a lot of communications, protocols, recovery and backup, getting to
other banks, verification, etc. All of which has no human interface
whatsoever.
------------------
<footnote 2>. Never mind the fact that the function keys are on the left and
keypad on the right, necessitating continual eye-hand shifts. The most
bizarre example I have seen is the amazing fact that my bank's drive-up ATM
has Braille keypad and a Braille instruction area? Braille on a drive-up?
But note that neither of these have anything to do with software but
everything to do with design.
--------------------
But how much of software has any human interface at all? Again, a minority.
Don't measure by the number of desktops --measure by software development
labor content. Here's some questions to set a perspective:
1. What organization is the largest (in terms of labor content and
employment) software developer in the world?
2. Where does Microsoft rank in terms of total number of software engineers
in the world? <footnote 3>.
--------------------
<Footnote 3> Let's not get into the hassle of temp employees versus genuine
MSFT employees. If they spend most of their time working on MSFT software,
I'm counting them as MSFT software developers.
--------------------
I don't have a recent figure, but two years ago, the answer to question #1
was Siemens and the answer to question #2 was that MSFT didn't even make the
top ten. They were someplace around 20th or so. If you asked a person in
the street the answers to those questions, MSFT would most likely have been
given for #1 and been ranked first for #2. Siemens is instructive because a
lot of that software is embedded. Much of it is of a highly technical nature
(e.g., in manufacturing control) where usability concerns are minor. Siemens,
a few years ago, announced that it was redefining itself as a software company
and that the hardware was merely a vessel for the software -- or words to that
effect. There are approximately 3 million persons world-wide who do software
development. All of MSFT doesn't make of 1% of that total. What are all
those others doing?
We should not let mythology and popular misconceptions such as "Microsoft is
the biggest software development organization in the world" or "Most software
development is on PC software." or "Most software has a usability issue." be
the guiding principle of software development and therefore of software
testing. Our concerns should be proportioned to the actual labor content
spent in software development because testing is (should be) a relatively
fixed percentage of that content. On that basis, it is very hard to see why
and how usability should play more than 2% role in software testing -- (15% of
15% and I am being very generous with that). And, as we'll see below, it
isn't even a testing concern in the first place.
3. Usability Isn't a Software Issue.
Having set a perspective on how important usability is in the general software
world, we should ask if usability concerns are a software issue at all? No.
The fact that these things are being implemented in software is almost
accidental. Back to the ATM. Are there any software issues (and therefore,
software testing issues) in the usability aspects of the ATM. Absolutely
none. There is nothing in the public ATM interface that can't be implemented
almost as easily in hardware. I bought a new car in January. There's a lot
of software in it. Is there a software testing issue in what I, as a user,
can see. Not much from a usability point of view. Sure there's usability
engineering in that interface -- I hope so. But is there is a software issue
in that interface and more specifically, a software testing issue? Again no.
If usability design has been done before the fact (as it should) then
usability behavior and appearance is just another feature set to be tested to
assure that the interface has been implemented as intended. Any competent
tester can do that and furthermore, if the desired behavior has been embedded
in a prototype, most of those tests can be automated.
Usability is all about design and should not be about testing at all. All
usability issues today can be explored and resolved before any software is
written. Not only can they, but they should be. That's what prototypes are
for. It isn't testing. It is exploration. It is doing sketches. It is
checking out concepts to see if they fly. It is no more a software testing
concern than a designer's exploration of using sort routine A versus sort
routine B. It is the fact that all such issues can be resolved on prototypes
whose code has nothing to do with the implementation code that makes it clear
that it is not a software testing concern.
4. It isn't a Testing Issue.
Making usability a testing issue is to condone bad design practices.
Unfortunately, such practices are common. We should object loudly and clearly
when bad human interfaces have been designed and when design groups expect us
testers to find that out and worse, to fix it. There should be no usability
issues whatsoever by the time software gets to testing. If there are, it is
often too late to do anything about it anyhow. I see usability as a testing
issue as bizarre as a data-structures testing group -- a hypothetical,
after-the-fact, group concerned with confirming that the design group had
picked the correct data structure. Or how about an architecture verification
group? Of all the factors that go into the creation of a viable software
product, only two are routinely relegated to discovery by quality control --
usability and performance. In both instances, we are dealing with a failed
design process. By allowing usability to be a testing concern, we are not
only condoning a fundamentally bad process, but we are abetting it, and worse,
legitimizing it.-- of course, only for that small part of that minority of
software that has a human interface worth talking about.
5. Alien Disciplines.
Let's look at UDA (usability design and assessment) as a professional
discipline and contrast it to testing as a discipline.
5.1. Base discipline: Software Testing's discipline is a subset of computer
science and software engineering. UDA's underlying discipline are the social
sciences, especially applied psychology. If ever there were two groups from
opposite sides of the campus, it is these. The fundamentally different base
disciplines means different training, different ways of thinking, and
certainly different prerequisites. The social scientist can only read and
understand a small fraction of the software testing literature. They simply
don't have the prerequisite training in mathematics and abstract modeling.
While I can see how to instill a mutual appreciation of each other (to the
minor extent that it is important for small parts of some software) I do not
expect social scientist to learn even elementary graph theory and I imagine
that they don't expect me to rise above the naive level of pop-psychology of
the Dear Abby ilk.
The biggest problem with two conflicting base disciplines is the erosion of
professionalism on both sides. We have software engineers doing a bad job
(usually no job) of human interface design and an even worse amateurish job of
assessment -- and equally incompetent people trained in the social sciences
blathering about superficialities and trying to constrain all of software
testing under not only behavioral testing, but under the specific small part
of behavioral testing that concerns the human interface. This lack of
professionalism perpetuates the often low status of testing <footnote 4> You
can't get respect if it is widely perceived that almost any kind of training
from psychology to law (and even software engineering) prepares you
professionally for software testing. Without clear professional prerequisites
there can be no recognition of a profession. The conflicting base discipline
is at the heart of this issue.
------------------------
<footnote 4> I don't know about the status of UDA, because that's not my
concern. I assume that the UDA people can look out for their own status. I
am concerned with the status of software testers.
-----------------------------
5.2. Technology.
The technology of UDA is video cameras, one-way mirrors, carefully crafted
questionnaires, psychological testing methods and associated statistical
methodologies, and lately, some recent software products that can be used to
capture user's behavior -- but never replay. It is a mostly manual technology
and until they can automate psychology, it will continue to be dominated by
manual/human methods. And most of us wouldn't call it a technology at all.
Test execution automation is meaningless for UDA because it completely ignores
what UDA is all about -- observing human behavior -- and if you take the human
out of the loop, what's left to observe? Test design automation is even more
meaningless for UDA.
I don't doubt that there can be technology that will make UDA more
effective-- and that technology may even be software based -- but it is clear
that it is a very different technology than that of software testing --
automated test execution, automated test generation, coverage tools, stress
testing tools, -- to name a few. The lumping of the two disciplines under
one roof means a continual conflict over what tools are needed -- and more to
the point, continual harangues on newsgroups and discussion groups over how
much of testing can and should be automated. A clear separation of the
disciplines will go a long way towards alleviating tool conflicts and tool
chaos, and putting to rest most discussions on how of testing should be
automated.
5.3. Objectives.
Totally different. Software testing is about finding bugs. UDA is about the
design of the human interface. Software testers try to break software -- to
see that it is built right. UDA specialists are concerned with seeing to it
that the right thing is built for a small part of the software. Negative
testing has little or no place in UDA -- in fact, the very concept of
positive/negative testing is meaningless until there is an implementation to
test.
5.4. Cross-Disciplines?
Very popular notion. "I'm not a discipline X person, I am an X-Y cross
disciplinarian." Let's look at this notion for a moment, and see where it
has gotten us, historically, in the development of technology.
1. To have a cross discipline you first need to have two disciplines to
cross. We are still struggling to get software engineering recognized as a
discipline, never mind software testing as a subset of that discipline. It is
far too early to start crafting a cross-discipline for testing and UDA --
e.g., software-psychology-engineering?
2. Cross discipline is a very hard road to travel. Mainly because most of
them are failures and/or fakes. The failed mathematician say's to the
engineer "You wouldn't understand, that's a mathematics issue." and to the
mathematician, "You wouldn't understand, that's an engineering issue." Cross
disciplines invite fakers, failures, and con-artists. That is why university
faculties have always been very wary of them. They do emerge (most don't),
but very slowly, with great difficulty, and over a very long period of time --
Anthropology is a good recent example and that only took 100 years. Forensic
medicine is another recent addition.
3. Real cross-disciplinarians do exist. It is easy to know when you have one.
They are trained and adept in both disciplines. Patent law is a good example,
where many practitioners have an undergraduate engineering degree followed by
law school. They contribute to peer-reviewed journals with learned papers in
both disciplines. They eventually have journals and conferences of their
own.
6. How Did We Get Here and Should We be Content With Where We are?
How did software testing get to be such a misch-masch? UDA, bug-hunting,
localization, performance, documentation, training, and a whole lot of other
junk tossed in? Because developers used it (testing) as a dump for all the
things they did not want to do themselves. The place to put all things, after
the fact, that should have been done all along, or before the fact (of code
creation). The first and most obvious is the unfortunate practice of lumping
a process control and management (Quality Assurance) in with a quality control
function -- testing. We keep saying that we can't test quality into a
product, but they still seem to be trying to do so. You more obviously can't
test usability into a product -- you can only design it in. I have seen
unfortunate organizations in which documentation was a major responsibility of
the software "test" group. The main reason we got into this mess is because
we allowed testing (or often misnamed "QA") to become the dumping ground for
all the process flaws. UDA is just one example of such mislocation. To
continue to allow this hodge-podge of alien disciplines and concerns to reside
under one roof, and to somehow, attempt to reconcile their conflicting
everything, is to condone a major process flaw -- and as I said before, to
abet it, and even worse, to promote it as the way things should be done.
We should not be content with that. We should fix process flaws, not support
them or glorify them and certainly not condone them under a vague, putatively
holistic, interdisciplinary rubric. Let software testers be trained and be
concerned with the testing of software -- all of software, in balance, and
not a superficial part of a minority of it. Let them fish on the software
side of the stream. Let the UDA people get out of the patch-up business
after-the-fact and put their efforts where it belongs -- before any software
is written -- to the extent that a piece of software has UDA concerns at all,
that is. Let them fish on the human interface side of the stream. And for
the moment and the foreseeable future, let there be no squabbling as to who
fishes in the middle, because if we waste our limited human resources in such
pointless conflicts, the fish will pass us by and all will surely go hungry.
Boris Beizer
-------------------------------------
Boris Beizer Ph.D. Seminars and Consulting
1232 Glenbrook Road on Software Testing and
Huntingdon Valley, PA 19006 Quality Assurance
TEL: 215-572-5580
FAX: 215-886-0144
Email direct: bbe...@sprintmail.com
Email (Forwarded): bbe...@acm.org, bbe...@ieee.org
Copyright, Boris Beizer, 1999)
------------------------------------------