Computer-Supported Per Review Systems

[Christopher Wolfe]

Dear CSPRED colleagues,

Ilya Goldin has kindly invited me to join this list, and I am writing
for your assistance with Computer-Supported Per Review systems. I am a
Professor of Psychology at Miami University (in Oxford, Ohio). I am
leading a group of faculty who wish to use Computer-Suppoted Peer
Review Systems in our university teaching (and possibly research). We
have developed some principles for use and some desired
specifications, and now we are considering candidate systems. SWoRD is
one that we are considering, but as our next step we would like to get
a sense of what's "out there." Thus I am asking for your help with two
things:

(1) identifying any and all Computer-Suppoted Peer Review systems,
ideally with web links, and
(2) your opinions of which is best and important things to consider.

Any help you might provide would be very helpful. Thanks! - Chris
Christopher R. Wolfe, Ph.D.

P.S. Since many of you are at Pitt I'll share that I earned my
doctorate at Pitt in 1989 and had an office in the LRDC. You can find
out more about me at: http://think.psy.muohio.edu/home/index.html

[Schunn, Christian Dieter]

Let me add my two cents, with obvious bias.

There many different kinds of peer-review options. I'm going to focus on systems for writing. I'm not sure whether you want to consider peer review of things other than writing. At CSPRED, we had many systems that focus on programming.

1) There are many, many research-based systems that are only locally used and not really ready for many other users. Upside: free for use, often cool features, possibilities for research collaboration. Downside: little user support, buggy, unknown longevity.

2) There are commercial systems bundled with freshman writing textbooks. With publishers consuming each other, I'm not sure which ones have what this year, but I imagine you can find that out quickly. Upside: integrated support of many aspects of writing together with peer review, adequate user support, relatively bullet-proof, decent interface. Downside: little support for research, still changing interface and features, optimized for freshman writing, requires textbook adoption.

3) Blackboard has some peer review features (and probably some blackboard competitors, too). Upside: peer review within integrated LMS, some user support, can use for peer review of anything. Downside: very awkward interface (like everything in blackboard), little support for research.

4) Turnitin.com has a peer review system (peermark; https://www.turnitin.com/static/products.html) that can be added to the plagiarism detection system. Upside: pretty good interface, good user support, broad library of rubrics. Downside: little support for research, extra cost.

5) Calibrated peer review (http://cpr.molsci.ucla.edu/). Upside: great for short writing to learn exercises, free, easily integrated into many kinds of classes. Downside: little user support, not great for longer learning to write assignments, can be slow because so heavily used, little support for research.

6) Sword v5 (http://sword.lrdc.pitt.edu/). Upside: free, some user support, possibilities for research collaboration, already setup for research data use. Downside: still changing interface and features.

> --
> -- Computer-Supported Peer Review in Education
> -- website: http://cspred.org
> -- discussion: http://groups.google.com/group/cspred
> -- email the group: csp...@googlegroups.com
> -- unsubscribe: cspred+un...@googlegroups.com
> --

[Christopher Wolfe]

Thank you for the detailed and very helpful reply, Christian. Best
wishes. -Chris Wolfe

[Ilya Goldin]

One addendum to Chris Schunn's summary is this: the suitability of any
given system will depend on your particular requirements. Even if all
you want is peer review of writing, systems will vary in terms of
instructor support, grading support, flexibility in terms of review
forms (and back-review forms), etc. If you're interested in research,
different systems will be easier to fit into different kinds of
research agendas.

Just for purposes of illustration - my own system, Comrade, was
written to support my dissertation research. This meant that I did not
implement some things that other systems take for granted, but I did
implement support for research experiments. This means that students
can be scripted to take different paths through the exercise,
including not only the peer review itself, but also pre-tests and post-
tests, surveys, and so forth.

Please feel free to share information as you go through your selection
process, and to ask questions along the way.

Best,
Ilya

On Jul 25, 1:42 pm, "Schunn, Christian Dieter" <sch...@pitt.edu>
wrote:

[Christopher Wolfe]

Great, Ilya I will take you up on this generous offer. -Chris

[Ed Gehringer]

Sorry for not responding earlier.  On Sunday, I was returning from Michigan by car.

 

Our system is called Expertiza, and has features quite different from most peer-review systems.

 

1.  It is oriented toward assignments where students pick topics from a list, with a limited number who can pick any particular topic.  The system keeps track of who has picked which topic.  This allows a large project to be separated into chunks that are manageable for a student or a team.  Often we assign reviewers to review several submissions on the same topic, to allow the reviewers to gain some expertise in that topic.

 

2.  Students may suggest topics for assignments using Expertiza, and then the instructor or TA can approve (or disapprove) each suggestion.  If approved, topics go into the list from which students choose.

 

3.  Expertiza supports team assignments.  All team members have access to the same "submission area."  Normally, individuals are assigned to review teams.  Team members can rate each other's contribution to the team effort, and these ratings can influence computed grades.

 

4.  Authors are allowed to give feedback to reviewers.  Author feedback can also be used in computing grades.

 

5.  "Metareviews"--third-party reviews of students' reviews--may be used as another technique for improving reviewing.

 

6.  Authors may resubmit at any time, and reviewers may "re-review" at any time.  We are working on a way to "automatically" give extra credit for doing extra reviews.

 

7.  Submissions may be either files (of any type), wiki pages, or URLs.  If wiki pages are used, students may be assigned "handles" to use in logging into the wiki, so their reviewers can't determine their identity from the wiki history feature.

 

8.  For long multipart assignments, like authoring a wiki textbook, where the different parts have different deadlines, the instructor can set up a single set of deadlines and input the precedence relationships between parts, and the system will calculate the deadlines for different parts.  This relieves the instructor of a lot of bookkeeping associated with a large project.

 

9.  "Everything" in Expertiza is rubric based.

 

If you would like to explore the system, or use it in your classes, we would be glad to work with you!

 

Thanks,

Ed

P.S.  I've never been at Pitt, but long ago, I was a research associate and lecturer at CMU.

>>> Christopher Wolfe <wol...@muohio.edu> 7/25/2010 11:45 AM >>>

[Christopher Wolfe]

Thank you, Ed. This looks really interesting. My colleagues and I are going to look things over, and we will, no doubt, be back in touch with you and the other CSPRED colleagues. Best wishes. -Chris