Assessment and evaluation options for this semester

[Alan Webb]

Hi everyone,

Have you thought about how participants in your course this semester will set challenges for themselves and others, share and celebrate accomplishments, and give and receive feedback on their successes?  If not, I just want to discuss some of your options and hear from you what you think you'd like to do for your course.

We are participating in piloting "badges" with the P2PU School of Webcraft this semester, so one option you have is to figure out a way for participants in your courses to earn badges.  We've done some thinking on this already you may want to hear about.  School of Webcraft may have some badges of their own, we may share some badges with them, and we may also have some badges of our own too if any of you all have interest in doing that.  

Quick run-down: badges are essentially digital certificates that can be put anywhere on the web that can be traced for authenticity back to peers or trusted reviewers who recognized that you have exhibited competence or mastery in certain strengths or skills.  For example, you could earn a badge saying that you are quite the community artist which you could put on your Facebook and LinkedIn portfolios and on your blog.  For SoW, they already have some hard skill badges in mind, like on Javascript and on designing for Web Accessibility, that will unique to them.  There are some badges we could also share with them are intended to recognize deeper skills like empathy, creativity, cooperation, etc.  They have some specific ideas about awarding badges for team players in courses or for veteran course organizers, for example, and we may both want to recognize people who showed themselves to be great facilitators or hosts in their courses or in local citizen circles.

For badges unique to SoSI, each of you may want to spell out specific badges participants in your courses may earn.  For example, I think that Emily, Tedd, and Ambassador McDonald could define a Conflict Resolution badge based on the ideas Ambassador McDonald outlined in the boy scouts merit badge book years ago.  If so, you could work with the Institute for Multitrack Diplomacy or others to outline possible challenges students could complete in order to earn that badge, and write out a brief rubric for peers to evaluate whether work they complete by the end of the course deserves a badge.  The Sustainability Studio also has a head start with eight competency areas ready-made to becomes badges; MSU has already done the work to define and develop rubrics for each of these.

As far as tools for assessment and evaluation, I think you all have three general options, from most basic to most advanced, though you are welcome to suggest others or mix and match from below.  I would like to hear what you think works best for you:

1) DIY.  Use google docs or any other word processor, any existing portfolio tool you like, or anything else you can dream up to let students set goals, track their accomplishments, get theirs classmates to vouch for them, etc. according to your course's needs.  If you want to do a Living Transcript exercise in your course (a more personal, thorough and retroactive portfolio of your changemaker competencies, looking back on the whole of your life, with some steps taken to get your friends to vouch for your competencies), we can give you a template to use for that.  In the pilot course last semester, we each all did a living transcript for ourselves and shared with each other for feedback at the beginning and the end of the course.  That was a great exercise that was liked by all the participants.

2) Options (2) and (3) are two different ways to use a new online tool P2PU and Mozilla are working on creating, by modifying a tool called OSQA (Open Source Question and Answer).  I have attached some wireframes to give you a rough idea of how that system might work for SoW (keep in mind: these wireframes may not reflect the final solution we end up with) and we have an opportunity to say how we might use it and outline some changes we would like to see to that tool for our use as well.  The basic idea is that they are turning "questions" into "challenges," "answers" into "evidence," requiring that comments be attached to "up votes" (thumbs-up), and building in some logic that basically says "if x number of up votes have been received for one 'answer' award that user a badge."

One very basic way we could use this tool would be to think of it as a big wall which all courses share for giving congratulations and for recognizing any small or large success we notice (for ourselves or others) throughout the semester.  I've requested that we be able to filter the view of that wall by School (SoSI at large) and by courses (see all threads tagged by "Education Politics").  We could just let users use this any way we want over the course of the semester and, then, at the end of the semester, look back and retroactively nominate people for badges and decide to actually award them (manually) if enough people in the comments "up vote" (click the thumbs up and leave a comment) supporting each nomination.  That's the most organic and loose option, but more of a coordinated approach between courses than option (1) would be.

3) A third, more advanced way to use OSQA, which is also slightly more rigid, would be similar to how SoW is giving the Javascript badge.  In this case each of you could set "challenges" from the get-go for participants to complete in order to earn certain badges you already think they may want to earn, and define a rubric for others to judge whether they should accept evidence that a person has submitted that they have completed that challenge.  

These challenges could originate from existing rubrics (e.g. MSU's sustainability competency framework) or these challenges could also originate by asking for suggestions of specific challenges (projects) from organizations with which participants may already be working (this is similar to the model Laura learned that KaosPilots uses in Denmark).  For example, Laura thinks that in the Women as Social Innovators course she may be able to get a local nonprofit working on women's empowerment to suggest some challenges to participants in that course.  If students who sign up for her course want to take those challenges on, they could be posted as the start of a thread on the OSQA "wall" and students can discuss the challenge, submit evidence for completing the challenge, and get a "badge" for completing it when they do submit evidence and get enough "thumbs up" from both the organization issuing the challenge and from their peers (with supporting comments).  We could also seek to get organizations like AshokaU to suggest and "sponsor" certain challenges.  I think we would only put challenges up if we think it is a rigorous, non-trival challenge, it is evidence of development of some important skill for a specific participant who wants to take on that challenge, and it actually creates some important outcome for the organization issuing it.

I know there is a lot to process here, so let me know what you think works best for you, and feel free to discuss here to give me a call to talk it out!

Also, if we have any specific requests for modifications we would like to see in OSQA so that it could better serve our purposes, we should propose those here now.  As I said above, we're requesting that we can filter the view of the larger environment by SoSI and by course, the ability to tag threads by competencies, and, if enough people want to go with option (3) in some way, we may want to add things like special user permissions for people whose approval is needed for certain badges and logic to automatically issue badges based on certain activity on threads tagged a certain way.  Anything else you would suggest?  (Please also note that we're not going to have the OSQA environment until 4-6 weeks from now.)

Thanks everyone!

Best,

Alan