An inconvenient thesis?
Yishay Mor
An inconvenient thesis?
Christine Elizabeth Wania's phd thesis questions the empirical evidence for the benefits of pattern languages in HCI:
For more than two decades much of the pattern language literature, within the field of Human Computer Interaction (HCI), has focused on the possible benefits pattern languages may provide, but there has been very little empirical work to support these claims. It has been suggested that interaction patterns or pattern languages in HCI may address some of the problems inherent in designing interactive systems by supporting reuse, capturing design knowledge, enabling the sharing of design knowledge, and facilitating communication among designers and users. This study examined the impact of a pattern language on the design of information retrieval interfaces, in terms of the quality of the interfaces and the time to design the interfaces. Participants created paper and pencil interfaces based on the given design task. Participants were exposed to either a pattern language, guidelines, or no structuring technique. There were no statistically significant differences between the three groups in terms of the quality of the interfaces and time to design the interfaces. The results of this study suggest that the value of pattern languages in HCI may not be in reuse, at the early stages of design, or in terms of the quality of the resulting designs, in domains familiar to designers. Although there was no apparent impact of the pattern language on the early stage designs, the results of a follow-up study suggest there is a significant correlation between the existence of patterns in commercial systems and the overall usability of those systems. Therefore, we suggest that we, as a community, very closely examine the current state of pattern languages in HCI before continuing to move forward. As a community, we need to shift our focus away from discussing the possible benefits of pattern languages and trying to build pattern collections. And instead, focus on trying to fully understand the value of pattern languages in HCI. In doing so, the HCI community, will then begin to see the benefits from all the great efforts in this area.
About time someone asked this question. I mean, I'll swear by my favourite patterns and pattern languages, defend them as if they where my children flesh and blood, but do we have any evidence, in the scientific sense? In a way, this brings back the discussion on validity, resonance and aggregation. Yes, patterns work great for those who believe in them. But shouldn't we aim higher?
___________________________
Yishay Mor, Researcher, London Knowledge Lab
http://www.lkl.ac.uk/people/mor.html
http://www.google.com/calendar/embed?src=yishaym%40gmail.com
+44-20-78378888 x5737
Christian Kohls
expressed critique there is the better we can state our arguments and
defend the pattern paradigm – or, if we cannot find sound arguments, we
have to admit that there are holes in our approach. But let me just
quickly react to what I gained from the abstract.
First, it draws conclusions from comparing participants using pattern
language, guidelines, or no structuring technique. So what? What I am
interested in is the quality of each of the chosen vehicles themselves,
i.e. is it a good pattern language? Are these good guidelines? Of course,
good guidelines are superior to bad pattern languages. So how can I be
sure that the quality of the varied material is of equal strength? There
are a lot of assumptions made in that experiment which are probably NOT
empirical supported as well.
Of course there are good and bad patterns and pattern languages. In the
same way as there are good and bad books, presentations, restaurants etc.
Now, if you read a bad book are you going to question the quality of books
in general? Sometimes the same story is better told in books, sometimes it
is better told in the movies. So, does any of these incidents question the
media formats?
In fact, not all HCI pattern languages are of good quality, unfortunately.
Many of the pattern languages have never seen a Writer’s Workshop and have
never been shepherded. More over, many of the so called HCI patterns are
not pattern descriptions at all but guidelines wrapped up in pattern
format.
Now, what is the empirical evidence for patterns? Why is it that Yishay
would swear on his favourite patterns and defend them as if “as if they
where [his] children flesh and blood”. For a simple reason I think: he has
EXPERIENCED the patterns, they are part of his world picture. To say that
the pattern I have in my mind are wrong would be to say that I have a
mental problem. The patterns I have in mind are there because of what I
have experienced. Just as I have patterns of trees, cars, houses etc. in
my mind. I have seen them. I have created my own picture of them. And yes
I can be wrong at some points and – even more common – I may be incapable
of properly communicate the patterns of my mind. It’s even hard for me to
express what, in essence, a table is for me. I know the design space but I
cannot describe it completely – my languages is not rich enough for that.
Can you proof that writing e-mails, do cooperate presentations, organize
meetings, writing contracts, build universities and schools, install
traffic lights, sell things on a market etc. are patterns? No. Yet we
would all agree that these are recurrent things (that is patterns) and
that they are designed. Therefore, it’s fine to describe them as patterns.
There are other ways of describing them. You don’t have to use the
literature format of patterns. The pattern format has its qualities
because it guides you to a good analysis and lets you ask all the
important W-questions. For example, you need to make the problem explicit
(which is not the case for guidelines). This is important because in many
cases the un-experienced designer is not aware of the problems that arise
in a certain situation. If you run an experiment in which you describe the
task/problem already, then it may well be that a guideline is as good as a
pattern description – because the problem is obvious, hence, one of the
key benefits of patterns (make problems explicit and reason about them)
does not account.
I am afraid that I am currently applying an anti-pattern: taking a
position about a thesis I have not even read. Maybe it is just because the
existence of patterns has been questioned so many times among some of
my/our colleagues. But to question whether there are patterns is to
question whether there are doors, cars, chairs, cell phones etc. All of
these things are: recurrent structure with invariant second order features
and variant first order features. They all are designed and solve
problems. A world without wheels and tables would be quite inconvenient.
I am not arguing here that patterns are valid per se. Oh, no! We all know
that we people have different capabilities of pattern recognition. Some
people are more skilled in seeing the recurrent parts. Some people are
more skilled in judging what kind of type a person, meeting, book,
conference etc. is.
Personally I think that good patterns as well as good design is a process
of evolution, of piecemeal growth. The good pattern descriptions (those
which solve hard problems and are easy to read + understand) will survive
and can even be improved in upcoming generations. Irrelevant patterns or
badly written pattern descriptions will die because nobody uses them. If a
pattern is used and it helps people in doing things in the real world,
then I consider this a stronger empirical evidence than an experiment. In
a way, experiments are the opposite of our pattern approach: instead of
isolating properties, patterns consider emergent qualities of whole
entities. But that’s another story. Also, let me assure you that I did
some experiments with patterns, so I am not all against experiments ;-)
-- Chris
"Yishay Mor" <yis...@gmail.com> schrieb am Dienstag, 7. Oktober 2008 um
02:35 +0100:
[ http://patternlanguagenetwork.org/2008/10/07/an-inconvenient-thesis/ ]
>
>An inconvenient thesis?
>
>
>
>http://idea.library.drexel.edu/handle/1860/2829 ] phd thesis questions
>http://patternlanguagenetwork.org/2008/08/21/validity-resonance-and-aggregation/
>]validity, resonance and aggregation. Yes, patterns work great for those
>
>
>
>___________________________
> Yishay Mor, Researcher, London Knowledge Lab
> [ http://www.lkl.ac.uk/people/mor.html
> [ http://www.google.com/calendar/embed?src=yishaym%40gmail.com
>]http://www.google.com/calendar/embed?src=yishaym%40gmail.com
> +44-20-78378888 x5737
>
>
>>
--
Christian Kohls
Projekt e-teaching.org
IWM – Institut für Wissensmedien
Konrad-Adenauer-Str. 40
72072 Tübingen
Tel. +49 (0) 7071 979-103
Mail: c.k...@iwm-kmrc.de
WWW: http://www.iwm-kmrc.de/
WWW: http://www.e-teaching.org
Till Schümmer
I only had time to quickly browse through the thesis. Here are my first
impressions on what I saw (sorry for the length of this mail...):
First of all, I think that it is really important that we have
scientific studies like this to validate specific patterns and the
pattern approach as such.
However, I see various problems in this study:
1. Are the patterns used in the study perfect patterns?
The qualtity of the patterns for sure has implications on the results
that can be achieved with the patterns. If this pattern language would
have been discussed at a EuroPLoP conference, there would have been
probably some comments on the pattern style. If you consider, e.g., the
first pattern on page 166 of the thesis, we could ask the author where
the real problem of the pattern is. The Why section lists some
requirements (or forces) but there is no real tension preesent in this
section. Good patterns should, however, first of all let the reader
experience that there is something going wrong in situations where the
pattern is not in place. Look, e.g., at the first sentence of the Why
statement:
"Users should be able to express their queries and narrow their search
in a number of ways not just one."
Well, what's the problem behind this requirement? What would happen if
there was only one way to express the query? I strongly believe that it
is important to raise the reader's awareness for questions like this.
This may then help to uinderstand the essence of good solutions which we
intuitively use.
2. Who is the target audience of the patterns
Many HCI patterns can pay off when they are used in a participatory
design process (for those of you interested in moredetails of my
understanding of such design processes, you may read the thesis "A
Pattern Approach to End-User centered Groupware Development"
http://www.pi6.fernuni-hagen.de/publ/diss_schuemmer.pdf ). In my
experience, end-users have a lot of domain knowledge but have not yet
thought abiout why computer systems are built the way they are built.
The General Serach pattern on p. 196 is an example for such knowledge.
Everybody knows the google search box and if I would ask a user to
design a search mechanism, the simple search field will probably come up
as a first solution. The search filed definetely is a pattern, but a
boring one if it is reduced to a problem solution pair like this (I
exagerate the pattern that's in the thesis):
Problem: Users need to find information.
Solution: Provide them with a textbox and a search button that will
execute the search.
I don't see the real problem in the description of the pattern here.
Probably, there's much more than summarized in "users need toi find
information". The forces give some more details here:
"Many searches will be broad, general searches therefore it is
appropriate to provide a general search on the initial interface. Many
users may prefer to perform simple searches. Also for novice users or
quick, broad searches it is appropriate to provide a general search."
I would have loved to see a deepoer discussion on the nature of
information seeking. Is it that the users may have specific questions
and that these questions are naturally stated in natural language? Is it
that the question in most cases contains the most important key words
and that a search for these keywords will lead to the information that
the user is looking for?
3. Where are the limitations in the patterns of the pattern language?
Let's continue with the previous exemple: Once users understand why the
google search field is so powerful, they are ready to understand its
limitations as well. When is it better not to use a general search? The
pattern should provide answers here (I'm not sure if I could answer this
question ad hoc).
4. Are the patterns too simple?
Again, it depends laregely on the target audience. But if you look at
the General Search, there are not many design alternatives shown in the
pattern text. So, will the serach field alsways be implemented in the
same way? If yes, it would no longer fit to Alexander's definition of a
pattern since he states that a pattern is implemented over and over
again without implementing it the same way twice. Having variation
points is an essential part of a pattern. Teaching designers and users
to understand these variation points will have the effect that they can
create interesting designs or at least have interesting design discussions.
5. Is the target audience the same as the test group?
If the patterns are written for end-users, it is not sufficient to
evaluate them with designers. As stated on p. 83 of the thesis, the test
group consisted of students in the field of information retrieval. Many
of them have hat HCI courses before. I have the iompression that many of
them have already looked at existing systems from a designer's
perspective. Thus, they probably have some experience to which the
patterns did not add much. Would the results be different if the study
also observed end-users or domain experts of in other disciplines who
could participate in a design process?
6. Long Term Effects
The experiment was rather short. As stated on p. 85, the design tasks
had an average design time of about 20 minutes. The designed UIs shown
on pp. 241-246 further show that it was a rther simple design task. But
how would the patterns affect the design of a complete IS? We have made
good experiences with student groups working over one term on a design
project using our pattern language for computer-mediated interaction
(http://www.cmi-patterns.org). They started to use a common language in
their discussions which eased design decisions.
This leads me to what I found promissing in the thesis: The most
important part of this work can IMHO be found on pages 126-141. Here,
the author creates new hypotheses what the real value of patterns may
be. The following quotes provide an intro:
p. 126: "From this simple thought experiment it seems that we should not
question why there was no affect of the patterns on the quality of the
designs produced. It seems rather reasonable (although we do not have
empirical evidence at this time to support this) to suggest that the
participants used the patterns that they had been in contact with while
using information retrieval systems in the past. Although the
participants may not be aware that they were using “patterns” or refer
to them as patterns, they may have been actually using patterns or a
pattern-like structure."
p. 128: "they may be useful as an evaluation technique, they may aid in
the capture and sharing of design knowledge, and they may help improve
communication between those involved in the design process."
On the following pages, the author states that patterns cfan be used as
a tool for better understanding a domain and the design that takes place
in this domain. They should be considered as a tool for reflection (in
the sense of Schön). I agree with all of this, however, it would have
been nice if the thesis itself could provide some kind of proof here...
So, there's much space for future studies :-).
P.S.: I started to write the mail shortly before my third son arrived on
last Wed. When his birth started, I suspended my PC to disk and now some
days later I found the mail again. I also saw that there have been
planty mails in this thread. Some of my issues have been taken up by
others as well. I kept them in the mail anyway. I really appreciate this
discussion! Maybe we should think again on how to provide proofs for the
different pattern areas (HCI, Community Design, Participation, ...).
Cheers,
Till.
Yishay Mor schrieb:
> http://patternlanguagenetwork.org/2008/10/07/an-inconvenient-thesis/
>
>
> An inconvenient thesis?
>
> Christine Elizabeth Wania's phd thesis
> <http://idea.library.drexel.edu/handle/1860/2829> questions the
> <http://patternlanguagenetwork.org/2008/08/21/validity-resonance-and-aggregation/>.
> Yes, patterns work great for those who believe in them. But shouldn't we
> aim higher?
>
--
=============================================
Dr. Till Schuemmer
FernUniversitaet Hagen
Department of Mathematics & Computer Science
Cooperative Systems
Till.Sc...@Fernuni-Hagen.de
---------------------------------------------
Learn more about patterns for
computer-mediated interaction:
http://www.cmi-patterns.org/
=============================================