Recruiting question
Chris Blow
Just had a quick question about the recruiting process that is in
place for the testing that is happening.
Please excuse the question if it is off mark -- I am not able to
participate as much as I would like (love) to this month.
My question: how are users being recruited and screened for testing? I
have found that this can be a critical aspect of the testing; if
someone who would not usually use the software is expected to have
opinions about the usability, they may start to invent scenarios and
opinions that are not genuine, and thus invalidate the study.
In my experience it can be really difficult to get a good set of users
for a test -- sometimes it takes a great deal of recruiting effort --
and this may not be feasible for the resource-constrained and
volunteer-based testing that is happening. But my concern is that the
results may be counter productive if the users are not the actual
target based for the software design.
I usually spot this red flag when people start responding with phrases
like: "if I did need to use this I would probably ..." or the user
takes on a god-like tone, pretending that they know how other people
will behave: "Oh you don't want to do this this way, people never will
do X or Y ... someone who is using this will definitely want to do A
and B."
In my experience users can not accurately predict their own behaviors,
much less the behavior of others.
Preferably all of the data would be purely behavioral, as attitudinal
studies require extremely large sample sizes.
So I just wanted to blurt out this concern -- again please excuse the
caution if it is misplaced given my role as a total non-involved
slacker thus far. :)
Great work on all of this, I am really so happy to see this coming out
of the camps.
Chris Blow
Hunter Whitney
Chris,
That is a great question and important to ask. I was going to check in with
you today but have been traveling. Your participation in this is very
welcome and there are several key areas that really could use the help.
Included among them is u-test scripts, a fast and easy system to capture and
summarize key aspects of the projects to help in the UX evaluation and
recommendation process, and recruiting people to both give the test and
test takers.
This is early on, we have been drawing upon the people who have shown up at
the camp. I'll go into more detail below about things we can start to to
do...I'm taking a break at a rehearsal dinner so it may not be perfectly
crafted...but in the mean time, we have been lucky to get some good
volunteer testers and for some of the low hanging fruit type UI questions
such as "do you understand what this site is about or what to do here" can
be meaningfully and usefully addressed by many people. At the SV camp
recently, we were fortunate to have some very interesting volunteer test
takers show up, including a Haitian expat Public Administrator who has been
living in the US for 20 years and wrote me today that she is heading back to
Haiti to help with the rebuilding efforts. It was admittedly, serendipity
but she was able to give some great user feedback and I hope may help us
recruit more key user types for u-testing.
This is very large, interesting an evolving topic. It is also not an exact
science and any would question anyone who said it was. Nevertheless there
are some approaches.
The background of the people who are being tested is crucial. A process
many I've uses and many people I know, is to develop a set of user profiles
or personas based on stake holder interviews. This is essentially a best
guess of the user segments. Then you look for a number of people who fit
into each of these segments. There are initial screening factors such as
familiarity with the site/app or other knowledge that might skew the
factors.
Ultimately, I think it would be great to do a combination of in person
testing at the Camps when we can and do remote testing as well. Surveys may
be useful and also some crowd sourcing tools. That is another big and
interesting topic.
I should also note that there is always a place for a skilled UX person who
can ask questions in one-on-one tests or, if done carefully and skillfully,
small groups as well. A good, experienced UX person can elicit some great
insights and ideas from the users being tested. That said, less seasoned
testers can still uncover serious usability problems and identify common
trends and patterns with some very simple questioning. With screening and
some some good judgment by the testers, some of your concerns about people
giving problematic feedback.
The camps are a bit raw and unpolished and fueled by a lot of willingness to
experiment and try new things and know that some of it will stick and some
wont. I honestly think this camp is a great opportunity to apply UX ideas in
the same spirit and to make the overall effort more effective. I hope you
stick with this and ask questions and be a part of offering some of the
answers too.
As I said, this response is not perfectly crafted, but I wanted to start
addressing your question. Back to the dinner...
Cheers,
Hunter