Usability Lab Report
Steve Lee
I've also attached the summary data without quotes as one user wanted
to review them. It's not a particularly good format but the numerical
results can be read. Full data will be privately distributed to the
team.
We had the following devices to test: Desktop PC (19"), ASUS
Transformer Android tablet (10"), iPad (10"), iPhone, Dell Inspiron.
All devices were used. and users were also invited to try their own
devices.
The Usability Lab went smoothly, though we had initial issues with
wireless connectivity, and even wired access failed. Fortunately we
had Ross' Mifi and my Android smartphone tethering. It could even be
said the slow access was a more realistic test.
We had 9 testers, all rating themselves as fairly or highly competent
smartphone users, except one who was not a user. For tablets 5 do not
use them, 3 were fairly/highly competent and 1 not very. When it comes
to netbooks 2 were fairly/highly competent and the rest are not users.
1 user was an active member of myExperiment, 2 have only explored it
and the remaining have never used it.
Testers were mostly familiar with websites with social aspects and
mobile apps, but somewhat less so for VREs with social aspects. They
had medium Knowledge of usability and accessibility best practice.
In very broad terms the responses fell in the easy to very easy range
for searching but tasks that required subsequent 'editing' steps such
as adding a tag or joining a group were found to be more difficult.
There was variation in how easy tasks were perceived amoungst the
testers, perhaps based on previous experience with App style
interfaces.. It appears that users soon overcame initial causes of
confusion as they became more familiar with the style (to be expected
due to repetition in UI). Particular problems occurred when editing;
having to enter an edit mode rather than directly changing items
caused some confusion.
In the test widgets, several of the tasks simply presented a message
alert. These was often flagged as errors by the testers. That should
have been explained to them as part of the initial orientation.
Some of the specific issues are:
1) slow responsiveness made worse by lack of activity feedback (eg a throbber)
2) Not obvious how to get to details from search results
3) Icons were expected to be links
4) no clear navigation between screens - especially back - some missed
the common home button idiom
5) Device UI features getting in the way - list overlapping info, auto
capitalisation of text
6) Logged in status could be clearer.
7) Not clear need to enter edit mode with 'Edit' to change tags etc -
prefer more direct editing (eg AJAX style UI)
8) Sometime search results need differing info to that shown (tags
when searching for a tag)
9) Some action buttons not clearly labelled
Steve Lee
Programme Leader (Open Accessibility)
OpenDirective http://opendirective.com
Ross Gardler
Thanks Steve, some issues to consider below.
Sent from my mobile device, please forgive errors and brevity.
Can you expand on this please. None of the actual tasks set raised alert boxes other than to inform users that edits were not being written to the database. Are these the "errors" reported?
>
> Some of the specific issues are:
>
> 1) slow responsiveness made worse by lack of activity feedback (eg a throbber)
Agreed. The "throbber" is, in fact, implemented in code but not working in practice. I've not verified why. However, I intend to change the way the templates update their data and I suspect the new approach will solve this problem (as well as make it work in Android 2.2 browsers).
> 2) Not obvious how to get to details from search results
I'm not sure how to solve this one. Click anywhere on the item Assn your go to the details screen. Adding something like a "more" link will help, but it is a waste of screen space. On desktops the cursor changes to a pointer when over the item, so there is visual feedback there, but with no equivalent Hoover state on touch screens it doesn't work on touch devices.
We could do something with CSS, but what?
> 3) Icons were expected to be links
Which icons were not links? As far as I am aware they all are. Is it possible to narrow this down to specific icons?
> 4) no clear navigation between screens - especially back - some missed
> the common home button idiom
The original design documents questioned whether breadcrumbs were valuable on mobile platforms, based on feedback from the initial user evaluation, leaving the implementation decision to the dev team. The mock-ups had no navigation built into them at all. I implemented the "home" button at the last minute when I realised we had no navigation other than the back button.
This, I believe, is a major oversight and I intend to improve this in the templates.
> 5) Device UI features getting in the way - list overlapping info, auto
> capitalisation of text
Which devices overlapped lists? I didn't see this in any of my item testing.
As for auto capitalisation, this is a hardware issue that I am not sure if we can control from HTML5. Certainly something to look into.
> 6) Logged in status could be clearer.
Agreed, and in fact should be according to the UI specification which differs from the mock-ups in this respect.
> 7) Not clear need to enter edit mode with 'Edit' to change tags etc -
> prefer more direct editing (eg AJAX style UI)
+1
This is something I am likely to implement in the templates as they move forwards in Apache Wookie.
> 8) Sometime search results need differing info to that shown (tags
> when searching for a tag)
Interesting, this one never occurred to me during development. Why does a user require further feedback that a tag exists when it had been searched for? Is it better to provide this feedback and thus have coherent results depending on different searches? should this information be provided for all search results, using up valuable screen space? How does the system know the user is searching for a tag? Should the search distinguish between different types of search - further cluttering the screen?
It's a difficult one that I suspect would Brigit from some extensive A-B testing.
> 9) Some action buttons not clearly labelled
I assume this relates to "ascending" rather than "forward" rather than there were no labels?
Thanks
Ross
>
> Steve Lee
> Programme Leader (Open Accessibility)
> OpenDirective http://opendirective.com
Steve Lee
<rgar...@opendirective.com> wrote:
> On Dec 6, 2011 11:57 AM, "Steve Lee" <sl...@opendirective.com> wrote:
>> In the test widgets, several of the tasks simply presented a message
>> alert. These was often flagged as errors by the testers. That should
>> have been explained to them as part of the initial orientation.
>
> Can you expand on this please. None of the actual tasks set raised alert
> boxes other than to inform users that edits were not being written to the
> database. Are these the "errors" reported?
Yes the 'errors' where the fact they got a 'not implememented' type
alert. I don't see it as a problem with the deign or usability - but I
should have explained better. I did say we are testing the user
experience and not the myExperiment functionality, but was only really
clear on this with Dave de Roure to set his expectations
>> 2) Not obvious how to get to details from search results
>
> I'm not sure how to solve this one. Click anywhere on the item Assn your go
> to the details screen. Adding something like a "more" link will help, but it
> is a waste of screen space. On desktops the cursor changes to a pointer when
> over the item, so there is visual feedback there, but with no equivalent
> Hoover state on touch screens it doesn't work on touch devices.
How about adding a small > icon on the right of the expanded content?
When I mentioned it before I think we were talking about in the header
which already has the +/- expansion indicator there so would be messy.
I found an example here with fixed lists but I think will work on an
accordion (perhaps shading the expanded content background)
http://jquerymobile.com/test/docs/lists/lists-formatting.html
>> 3) Icons were expected to be links
>
> Which icons were not links? As far as I am aware they all are. Is it
> possible to narrow this down to specific icons?
Hmm I can't reproduce on trunk code and so now don't understand the
comment. I thought it was the thumbnails in the search results.
>> 4) no clear navigation between screens - especially back - some missed
>> the common home button idiom
>
> The original design documents questioned whether breadcrumbs were valuable
> on mobile platforms, based on feedback from the initial user evaluation,
> leaving the implementation decision to the dev team. The mock-ups had no
> navigation built into them at all. I implemented the "home" button at the
> last minute when I realised we had no navigation other than the back button.
>
> This, I believe, is a major oversight and I intend to improve this in the
> templates.
+1
>> 5) Device UI features getting in the way - list overlapping info, auto
>> capitalisation of text
>
> Which devices overlapped lists? I didn't see this in any of my item testing.
The following were mentioned specifically
* landscape on iPad couldn't see save button ( I assume OSK hid it)
* Android OS drops an autocomplete dropdown over the login box when
typing (this is surprising as it should position it not to obscure the
edit control)
> As for auto capitalisation, this is a hardware issue that I am not sure if
> we can control from HTML5. Certainly something to look into.
iOS Safari is the culprit here (just daft). It seems there is a user
'auto caps' option. Devs can use the autocapitalize='off'' but it's
not part of HTML5 so won't validate. HTML5 does specify
autocomplete="off". They should work on both the individual input tags
or the form tag.
>> 8) Sometime search results need differing info to that shown (tags
>> when searching for a tag)
>
> Interesting, this one never occurred to me during development. Why does a
> user require further feedback that a tag exists when it had been searched
> for? Is it better to provide this feedback and thus have coherent results
> depending on different searches? should this information be provided for all
> search results, using up valuable screen space? How does the system know the
> user is searching for a tag? Should the search distinguish between different
> types of search - further cluttering the screen?
Yes I had all those thoughts too. Each user might want a different set
of fields. Providing the tag you search for is really just
confirmation and moot point. I don't have a good answer yet.
> It's a difficult one that I suspect would Brigit from some extensive A-B
> testing.
>
>> 9) Some action buttons not clearly labelled
>
> I assume this relates to "ascending" rather than "forward" rather than there
> were no labels?
Yes plus some testers seem confused about mark and edit. Not sure why
other than not tottaly explicit (mark what?). Most likely is the need
to enter edit mode.
Steve
Ross Gardler
> On Tue, Dec 6, 2011 at 12:46 PM, Ross Gardler
> <rgar...@opendirective.com> wrote:
>> On Dec 6, 2011 11:57 AM, "Steve Lee" <sl...@opendirective.com> wrote:
>>> In the test widgets, several of the tasks simply presented a message
>>> alert. These was often flagged as errors by the testers. That should
>>> have been explained to them as part of the initial orientation.
>>
>> Can you expand on this please. None of the actual tasks set raised alert
>> boxes other than to inform users that edits were not being written to the
>> database. Are these the "errors" reported?
>
> Yes the 'errors' where the fact they got a 'not implememented' type
> alert. I don't see it as a problem with the deign or usability - but I
> should have explained better. I did say we are testing the user
> experience and not the myExperiment functionality, but was only really
> clear on this with Dave de Roure to set his expectations
OK, so this was people going off-piste - fair enough.
>>> 2) Not obvious how to get to details from search results
>>
>> I'm not sure how to solve this one. Click anywhere on the item Assn your go
>> to the details screen. Adding something like a "more" link will help, but it
>> is a waste of screen space. On desktops the cursor changes to a pointer when
>> over the item, so there is visual feedback there, but with no equivalent
>> Hoover state on touch screens it doesn't work on touch devices.
>
> How about adding a small > icon on the right of the expanded content?
> When I mentioned it before I think we were talking about in the header
> which already has the +/- expansion indicator there so would be messy.
> I found an example here with fixed lists but I think will work on an
> accordion (perhaps shading the expanded content background)
>
> http://jquerymobile.com/test/docs/lists/lists-formatting.html
Good suggestion, that should be workable.
>>> 3) Icons were expected to be links
>>
>> Which icons were not links? As far as I am aware they all are. Is it
>> possible to narrow this down to specific icons?
>
> Hmm I can't reproduce on trunk code and so now don't understand the
> comment. I thought it was the thumbnails in the search results.
Perhaps it was the slow network and lack of visual feedback (reported
separately). They clicked on it but say no response then clicked on
the title which appeared to work.
>>> 5) Device UI features getting in the way - list overlapping info, auto
>>> capitalisation of text
>>
>> Which devices overlapped lists? I didn't see this in any of my item testing.
>
> The following were mentioned specifically
> * landscape on iPad couldn't see save button ( I assume OSK hid it)
> * Android OS drops an autocomplete dropdown over the login box when
> typing (this is surprising as it should position it not to obscure the
> edit control)
Thanks, I'll test these and see what can be done.
>> As for auto capitalisation, this is a hardware issue that I am not sure if
>> we can control from HTML5. Certainly something to look into.
>
> iOS Safari is the culprit here (just daft). It seems there is a user
> 'auto caps' option. Devs can use the autocapitalize='off'' but it's
> not part of HTML5 so won't validate. HTML5 does specify
> autocomplete="off". They should work on both the individual input tags
> or the form tag.
OK. That's valuable feedback then. Sometimes platform specific stuff
means we can't be standards compliant. This needs to be rolled into an
implementation decision.
As an aside, I wonder if things like PhoneGap/Callback would fix this
for us when generating a native app. I'll experiment one day.
>>> 8) Sometime search results need differing info to that shown (tags
>>> when searching for a tag)
>>
>> Interesting, this one never occurred to me during development. Why does a
>> user require further feedback that a tag exists when it had been searched
>> for? Is it better to provide this feedback and thus have coherent results
>> depending on different searches? should this information be provided for all
>> search results, using up valuable screen space? How does the system know the
>> user is searching for a tag? Should the search distinguish between different
>> types of search - further cluttering the screen?
>
> Yes I had all those thoughts too. Each user might want a different set
> of fields. Providing the tag you search for is really just
> confirmation and moot point. I don't have a good answer yet.
Valuable usability feedback again I think. Definitely a case of A-B
testing on a much larger group of users before we could make a good
call on that.
>> It's a difficult one that I suspect would Brigit from some extensive A-B
>> testing.
>>
>>> 9) Some action buttons not clearly labelled
>>
>> I assume this relates to "ascending" rather than "forward" rather than there
>> were no labels?
>
> Yes plus some testers seem confused about mark and edit. Not sure why
> other than not tottaly explicit (mark what?). Most likely is the need
> to enter edit mode.
I can understand "mark" being confusing, it's entirely
non-descriptive. It makes sense for users who are familiar with
myExperiment. "reverse" and "forward" do seem like strange choices
since it's more common to see "ascending" and "descending" but then
again, they are less "geeky" terms so I suspect in a broader user test
we'd find more acceptance of the "forward/reverse" option. Not
understanding "edit" seems strange, although they might have been
wondering "what am I going to edit?"
Thanks for your further comments.
Ross
--
Ross Gardler (@rgardler)
Programme Leader (Open Development)
OpenDirective http://opendirective.com
Steve Lee
> On 6 December 2011 14:39, Steve Lee <sl...@opendirective.com> wrote:
>>>> 3) Icons were expected to be links
>>>
>>> Which icons were not links? As far as I am aware they all are. Is it
>>> possible to narrow this down to specific icons?
>>
>> Hmm I can't reproduce on trunk code and so now don't understand the
>> comment. I thought it was the thumbnails in the search results.
>
> Perhaps it was the slow network and lack of visual feedback (reported
> separately). They clicked on it but say no response then clicked on
> the title which appeared to work.
That's plausible
>>> As for auto capitalisation, this is a hardware issue that I am not sure if
>>> we can control from HTML5. Certainly something to look into.
>>
>> iOS Safari is the culprit here (just daft). It seems there is a user
>> 'auto caps' option. Devs can use the autocapitalize='off'' but it's
>> not part of HTML5 so won't validate. HTML5 does specify
>> autocomplete="off". They should work on both the individual input tags
>> or the form tag.
>
> OK. That's valuable feedback then. Sometimes platform specific stuff
> means we can't be standards compliant. This needs to be rolled into an
> implementation decision.
Actually, another quick look indicates you need to use
autocomplete=off (HTML5 and android) plus autocorrect="off" and
autocapitalize="off" (iOS). Messy
Will be needed for many fields - espec email, userid and I guess password
> As an aside, I wonder if things like PhoneGap/Callback would fix this
> for us when generating a native app. I'll experiment one day.
That would be good - or JQM could potentially.
Steve