Remote testing Tools

[Gavin Wye]

Hello

I'm looking at the following tools for conducting some remote (un-
moderated) testing

http://www.loop11.com/
http://www.whatusersdo.com/home.php
http://www.webnographer.com/

Loop 11 looks good but doesn't do video of the tests as far as I can
see.
What users do does video but doesn't do recruitment
I haven't had a chance to have a detailed look at webnographer yet
(Sorry James!)

Does anyone have any recommendations. Here's what I'm looking for.

Secure access (via login) <-- this is very Important
An introduction to the test
Free play testing (and record/report on output)
Details of goal/task based testing scenarios (and record/report on
output)
Allow users to feedback on site via:
· survey
· forum
· email
Allow users to ask for Help (some kind of Community Management) I'm
probably going to plug in Get Satisfaction for this.
Is there anything else I’ve missed?


Gavin
@gavinwye

[David Jarvis]

Hi Gavin

I have used What Users Do a few times.

• They do have a panel you can recruit from.
• Lee who is the MD is very helpful on getting bits done that aren't necessarily shown as part of the service - eg we were able to set up a post test q'aire.
• It's quick and easy to get the video out to edit down highlights for yourself

However I am hesitant about the value of unmoderated remote testing, because in our tests people didn't engage as much with the sites as they would have done in a moderated test, plus there's no opportunity to do in-test follow up questions to really extract the insights and "why did you do/think that" out of the participants.

Still for quick and dirty (and cheap!) feedback on different interface styles I think there is still some value to be had.

HTH

DJ

 
 

---
Where I'm at:
::: Call: +44 (0)7814 931 305
::: Connect: uk.linkedin.com/in/dcjarvis
::: Read: dcjarvis.posterous.com
::: Stream: twitter.com/dcjarvis
::: Listen: soundcloud.com/dcjarvis

---

From: Gavin Wye <wyeg...@gmail.com>
To: UX Brighton <ux...@googlegroups.com>
Sent: Thu, February 24, 2011 10:43:43 AM
Subject: [UXBRI] Remote testing Tools

--
You received this message because you are subscribed to the Google
Groups "UX Brighton" group.
To post to this group, send email to ux...@googlegroups.com
To unsubscribe from this group, send email to
uxbri-un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/uxbri?hl=en

http://uxbrighton.org.uk/
http://twitter.com/uxbri

[Harry Brignull]

Agree with David - you could probably get away with using whatusersdo for this. You'll need to set up your own panel containing users that match your specs - which I believe has a one-off cost. 

Call Lee Duddell, he's very helpful. 

Of course, webnographer is for quant unmoderated studies with no video output. If that's what you're after then contact James/Sabrina!

Harry

--
Dr. Harry Brignull
User Experience Consultant
http://www.90percentofeverything.com
+ 44 (0)7598 239869

[James Page]

Hi Gavin,

Thanks for posting the question. Our website is needing an update, and we have been very busy recently, but have hired new staff so should be able to update it soon. 

To try to answer your question there are two types of remote testing. Moderated and UnModerated. What approach you take depends on your research question. 

Loop11 and Webnographer falls into the UnModerated category. WhatUsersDo fits kind of half way between the two. The Moderated, and WhatUsersDo approach means that you have to go through each response to see where the issues are appearing. 

The other approach is to use metrics. Loop 11 collects metrics. Time on Task, pages visited etc. Where Webnographer differs in that it also collects the metrics, then it uses statistical methods to uncover where the issues lie. We are more expensive than Loop11, but you will get lots of hand holding, and metrics that we believe that you can take actions on.  

Where we pride ourselves is helping you identify what are the issues that people are/will experience, and the root cause. At the moment we have experimental playback, but you would not want to sit through 50 to 1000 user sessions. The purpose of the Webnographer playback function is to be able to show a client where somebody is falling over, and not for discovery where the issues lies. 

If you mean by secure, that the site been tested needs to be kept off the Internet (Alpha Release/Intranet) or that the user has to log in, we often do those tests, but requires more set up time or our side.

In answer to David Jarvis´s interesting points, I will answer then in relationship to Webnographer. 

because in our tests people didn't engage as much with the sites as they would have done in a moderated test,

With  Webnographer we are trying to model peoples real behaviour on the web. 

What we have found triangulating between web stats, Webnographer, and lab testing is that participants spend about 2 to 4 times longer in the lab than in either Webnographer, or on the real site. Depending on the number of participants Webnographer will have a margin of error from +/-4% to +/-15%. The more time people spend on a task the higher the task completion rate is. Reduce task times, and number of errors increase. We can counter the lack of time one participant spends by having more participants. We normally aim for between 50 to 1000 participants. 

 plus there's no opportunity to do in-test follow up questions to really extract the insights and "why did you do/think that" out of the participants.

This is built into Webnographer.

Hope this helps

All the best

James

[Gavin Wye]

James

Thanks for all that very informative, and food for thought.

Gavin