|User testing of P2PU.org - results||Stian Håklev||6/16/12 4:31 PM|
As discussed on p2pu-dev earlier, inspired by the Stanford HCI class, I decided to do some user testing on P2PU.org. So far I've only invited one friend, but it still gave a lot of useful feedback.
I did a detailed write up of how I did the study and my findings and recommendations here: http://reganmian.net/wiki/User%20Study%20P2pu
The tl;dr for those who don't want to read that (but please visit :)) is
Note that this user study mainly looked at how brand new users approach P2PU, find courses, sign-up etc. The user did not actively participate in any courses on P2PU, so there is a lot of functionality on the course pages which was not tested in detail.
I hope this information is useful, and I would be happy to discuss how we could address some of the problems that cropped up. I am planning to do more studies like this as well (and it would be cool if others did it, and shared their findings).
In addition to this kind of user studies, I think it might be interesting to study the click-through data from our site. Do we currently have Google Analytics? Would it be possible for me or others interested in studying usability to get access? It would be interesting to see how often different functionality is used, for example how often people go to Full description, or click on an activity feed from the front page, use the course listing on their profile page to go to their courses, etc.
http://reganmian.net/blog -- Random Stuff that Matters
|Re: [p2pu-community] User testing of P2PU.org - results||Jos||6/17/12 10:28 AM|
I've done a bit of this myself too; it's fun but really hard to be
just watching without being able to get your hands on the keyboard or
give some clues! :)
So I gave a new user the task of "signing up for the App Inventor
challenge". It took him about 8 minutes to finally start the first
task. That's a long time!
Google brought him into the tasks list straight away, so I asked him
to go to the home page and start from there. This particular challenge
is a community pick, and it is the only way to find it in the learn
page (that and view all).
The user spent about 3 minutes going through the different options in
the search column. He couldn't understand why the schools were so
prominent if he doesn't know which school is running the course he
wants. Also tried most tags that thought fit, but none worked. Tags
didn't make sense at all (this has come up before, but if you see an
18 beside 'html' you expect 18 courses and not 4).
The user wanted to go back to the homepage, but clicking the P2PU logo
would take him to a strange page that finally he recognised as 'the
dashboard'. He also clicked on his user name a bunch of times, which
takes you… nowhere. He was not able to go back to the front page he
saw when logged out.
He also clicked in the 'chat' link, didn't make any sense to him and
went back to do something else.
In the get involved page, he ended up in meetup.com, after clicking
the 'Join or start a Meetup' thinking that it was the way of starting
his own group.
Finally went back to google to find the challenge again.
Clicking on 'Start the Challenge' sent him to log in, Sign up in this
case. Although he had just signed up for P2PU, he was also expecting
to sign up for the challenge. Clicking 'Start the Challenge' again was
not indication enough that he was already in, so he was a bit lost for
a while. (I can see this issue in the previous message from Stian
As a general comment, he considered that schools have way too much
weight on the UI (which is pretty much all you can see in a new
profile with no activity messages), and he was not interested in
schools, and in fact, the challenge he was after is not in a school.
The good news is that he found the site really interesting and is
thinking about starting a group! :)
> You received this message because you are subscribed to the "P2PU Community"
> group. Please stick to the ground rules:
> Specific topics such as research, web development and course design are
> discussed in separate working groups:
|Re: [p2pu-community] User testing of P2PU.org - results||Jessy Kate Schingler||6/17/12 12:04 PM|
wow these are so interesting guys, thanks so much for doing them. very valuable feedback and so hard for those of us who are used to the site, to see where these issues might arise. --
|Re: [p2pu-community] User testing of P2PU.org - results||Stian Håklev||6/17/12 2:00 PM|
Hi Jos, awesome work. I'm also confused, what does 18 after HTML indicate, if it's not the number of courses?
I also wish there was a way of quickly filtering courses based on a word (search by title and short description maybe). RIght now the only option is the custom Google search, which brings up links to all the content on the site...
Alison had a video about finding courses which was private, Alison have you fixed that, I'd be interested in seeing it.
So what's a good way for us to move forwards with fixing some of these issues? Some of these seem to be pretty obvious bugs, for example the fact that selecting a language in the language toolbar doesn't automatically delesect "all languages", but others might need some mocking up etc? And maybe we even need to user test the mockups, wire frames etc? What is the bottle neck? Is it the design work, or is it the coding etc?
|Re: [p2pu-community] User testing of P2PU.org - results||Jos||6/17/12 2:10 PM|
Zuzel's explanation about tags in this thread:
It does make sense from a dev point of view, but it has to be improved
for the final user.
Search should go into the priorities list discussed in the dev list in
the last couple of weeks.
|Re: [p2pu-community] User testing of P2PU.org - results||Vanessa Gennarelli||6/17/12 9:05 PM|
Stian and José--very interesting findings from both of you. José I would love to see screencaps of where your user went. I like the structured activity of "sign up for a particular challenge."
We should be encouraging this kind of design and usability research frequently. As we begin to establish a home for research projects, design research should also have a space. Your idea of usability bug tracking is a good one, Stian, but I'd also like to engage the community around design problems.
Ideas for housing current design research:
-an open googledoc folder
|Re: User testing of P2PU.org - results||karen fasimpaur||6/18/12 10:04 AM|
Awesome, Stian! Thanks. I think broader usability testing will be really, really useful.
|Re: [p2pu-community] User testing of P2PU.org - results||Jessy Kate Schingler||6/18/12 10:24 AM|
this ties perfectly into the discussion some of us have been having about creating more of a research/project space at p2pu. so, really curious what you guys think would be good for this.
the goal is to provide a place to house projects related to (of, or by) p2pu, that don't necessarily fit into the course or challenge model (like this!).
the solution being thrown around is a wordpress multi-site install. this could work in a few ways. different "projects" or teams (user studies group, research group, design group) could create spaces using wordpress, and via wordpress could create and manage their own posts (blog) and static pages.
similarly, people like corbin's DIY guide to open learning could host his efforts there, the HCIL and michigan research projects could create spaces too.
my concern about this approach is that it still feels like you have to somehow be an "official" project of p2pu. since so many open learners are also experimenters, i liked the idea that it should be almost as easy to create a p2pu project space, as it is to create a course/challenge/study group. but, maybe that's jumping the gun - this would let us start simple with real needs we know exist now. and, maybe we CAN make it easy, by making new project creation open to all if we so decide.
would love thoughts on this!
|Re: [p2pu-community] User testing of P2PU.org - results||Alison Cole||6/18/12 10:48 AM|
|Re: [p2pu-community] User testing of P2PU.org - results||Vanessa Gennarelli||6/18/12 12:44 PM|
Welcome to the community, Rodney :)
Jessy, I wonder if we can't make this a learning experience--I actually think a User Testing Challenge might be quite good. @Stian, you've got experience in this area and your data is quite thorough--would you take the lead on such a project?
No doubt that we need to plan for projects on P2PU-- in the meantime, I'd love there to be a place folks can learn how to user test, and have P2PU reap the benefits of the testing.
|Re: [p2pu-community] User testing of P2PU.org - results||Jessy Kate Schingler||6/18/12 1:19 PM|
well, obviously it is up to stian :) BUT... IMHO there is a concern that it's a bit of an unnatural fit, and adds unnecessary overhead. are we adding this overhead just for the sake of applying "our" model to something?
to do those user tests is itself work, and it would be great to have people doing more of them.... so to then require that those doing such user tests participate in/support a social experience on top of that effort, means that either we are asking even more of (some of) them, and/or we are likely to have a challenge that's a bit of a ghost town.
but this is more a general comment about the importance of having other types of outlets for learning activities and studies at p2pu :) as opposed to necessarily a point about the user studies (which i am only participating in as a consumer).
|Re: [p2pu-community] User testing of P2PU.org - results||Jos||6/19/12 7:58 AM|
I agree with Jessy here. A usability challenge would be great but it's
a lot of work. Just having a bunch of tasks without properly adding
all the history of why you are supposed to do certain stuff wouldn't
really work. Adding all that history would be a 6 weeks course, at
I realise the old pads are very limited and they wouldn't "scale" to
hold one of these new 'projects', but it could be a start. A
multi-site wordpress install needs maintenance, and users needs
accounts, which are not connected to their P2PU accounts, and it feels
official, and it sounds messy in general.
|Re: [p2pu-community] User testing of P2PU.org - results||Philipp Schmidt||6/19/12 8:56 AM|
(1) How to make sure that the results from UX testing will influence the site development process. This is something I've discussed with John and he will send an update on how the priority setting process works and what the most efficient way is for anyone to participate in the design and development process, so we don't loose this excellent feedback.
(2) A place for P2PU projects and working groups (we discussed a Wordpress Multisite). Examples are "P2PU Ambassadors", "Learning & Assessment research conducted by P2PU", "Schools Working Group", "Badges working group" - think of these as the different departments of the P2P University. Some of them will map to funded projects. The goal is to provide a space where they can post updates, and invite more people to get involved. Right now, all this great work that is being undertaken at P2PU is completely hidden. The best analogy I can find for these types of projects are the Open Knowledge Foundation projects:
(3) I am also a big fan of supporting learning experiments, but that's a different scenario and will require much more diverse technology in cases where p2pu.org does not offer the right set of tools. It would be hard to provide a lot of support for these projects at the moment, and realistically we will have to let this emerge a little on the basis of individual ownership and initiative at least for now.
|Re: [p2pu-community] User testing of P2PU.org - results||Stian Håklev||6/19/12 10:50 AM|
I like how you summarize this. I wanted to chime in earlier on some of the comments, but this is a good time to say it - to me, the main motivation for all of this is to improve the usability of P2PU.org on a short-term basis. I find usability and HCI interesting (which is why I am taking the Stanford HCI class), and it's something that would be interesting to discuss from a theoretical perspective, but right now I am far more concerned about getting our site to be more welcoming and user-friendly, especially since we'll probably get a bunch more people since the Chronicle article.
This is exactly why I asked in my previous message about how we can push this further - how can we non-coders (when it comes to lernanta I count myself as that) best assist the tech team with making sure some or most of these findings are addressed? On the one hand, we want to be as useful as possible to the tech team, on the other hand, clarity from the tech team on how priorities are set, what is being worked on, etc, will be motivating. If I spent a lot of time doing usability testing, and found that none of my recommendations were implemented, that would be very demotivating. :)
PS: Still wondering about access to Google Analytics, and whether we could use statistics from user click tracking to say something about which elements are most useful, which "paths" users follow, etc.
|Re: [p2pu-community] User testing of P2PU.org - results||Jessy Kate Schingler||6/19/12 11:19 AM|
we do have analytics. and, once we set up the regular database data releases, it will *freaking awesome* to do comparative analysis of behaviours over time as well.
philipp or john can add you to the google analytics account, i am sure! in fact i think mining that data (more than the cursory analysis gogole provides) is another way to provide really useful data to the dev/design teams. but, i also want to emphasize how kick ass the usability testing you and jos did is-- it's something that isn't getting done by anyone else, so i think getting more data points on that would be awesome too.
|Re: [p2pu-community] User testing of P2PU.org - results||Jessy Kate Schingler||6/19/12 11:31 AM|
philipp, i also think this is an excellent summary!
on points 2 and 3, if people like the multi-site idea, why don't we make one of the working groups a "projects" working group. it can at least be a stop-gap home for those who actively seek out a home for their experimental learning projects? that will also give us a sense of demand for that type of space, as well as the chance to observe how people use it.
talking with vanessa today, she pointed out that there will be SOME amount of bootstrapping to fill out content on such a site. so, there will be some fleshing out to do and i hope the respective "departments" will be willing to help populate the text/images/about page etc.!
i agree with jos' point that the multi-site direction feels more "official" and thus less lightweight for people. to philipp's point, i guess this is basically about triaging (providing something more than the etherpads because that's not QUITE enough), but less than some ideal customized development effort :).
if people agree, i had offered to do the install and administer the site. but its great to get feedback first and see how people feel about it and where they would/wouldn't use it, major concerns etc. and definitely open (natch, very interested in) possible alternative solutions...
|Re: [p2pu-community] User testing of P2PU.org - results||Philipp Schmidt||6/19/12 11:15 PM|
makes a lot of sense. +1
can we call it experiments? or lab?
"projects" is very broad and would be ambiguous (the ambassador project is also a project, but that's not what you are talking about)
|Re: [p2pu-community] User testing of P2PU.org - results||Jessy Kate Schingler||6/19/12 11:19 PM|
True, good point. Experiments or lab works.
|Re: [p2pu-community] User testing of P2PU.org - results||Alison Cole||6/20/12 10:00 AM|
Perhaps y'all can capitalize on some time in Berlin and do a few UX testing sessions aimed at generating quick-fix solutions? I would participate from home for sure.