reciprocity in peer review

[Ilya Goldin]

Dear all,

Do you know of any literature dealing with reciprocity in peer review?
By reciprocity, I mean that authors may be tempted to give high back-
evaluations to reviewers that give the authors' works high ratings,
and low back-evaluations in response to low ratings from reviewers.

Thanks in advance,
Ilya

[Ed Gehringer]

Hi Ilya,

 

What do you mean by "back-evaluations"?  I think you are saying that the authors rate the reviewers who rated them.  While I don't know of any research myself, I can ask this question on the POD list, which includes many faculty developers interested in various kinds of assessment.

 

If that is what you mean, my practice has been that a third party evaluates the reviewers ... which seems to evade the issue.

 

Thanks,

Ed

>>> Ilya Goldin <ilyag...@gmail.com> 10/22/2010 2:01 PM >>>

--
-- Computer-Supported Peer Review in Education
-- website: http://cspred.org
-- discussion: http://groups.google.com/group/cspred
-- email the group: csp...@googlegroups.com
-- unsubscribe: cspred+un...@googlegroups.com
--

[Ilya Goldin]

Yes, Ed, that's what I mean, reciprocity in the context of e.g., authors telling reviewers whether the feedback was helpful.

Any pointers would be appreciated.

Thanks,
Ilya

[Kwangsu Cho]

Ilya

Here is the one I wrote a few years ago.
http://portal.acm.org/citation.cfm?id=1769477

Hope this helps
Kwangsu

[Ilya Goldin]

Kwangsu, very nice paper. Thanks for pointing that out. Just to
clarify, in Table 1, the column titled "Peer Review Rating" is the
same as the row titled "Individual Peer Ratings", yes?

For completeness, I should mention that Kwangsu's paper cites Lin et
al. for noting the reciprocity phenomenon.

Sunny San-Ju Lin, E. Liu, and Shyan-Ming Yuan, “Web based peer
assessment: attitude and achievement,” IEEE Transactions on Education,
vol. 44, no. 2, p. 13 pp., 2001.
http://nctur.lib.nctu.edu.tw/handle/987654321/24508

Both of these papers are now in the CSPRED Zotero library.

Best,
Ilya

On Oct 22, 7:02 pm, Kwangsu Cho <ch...@missouri.edu> wrote:
> Ilya
>

> Here is the one I wrote a few years ago.http://portal.acm.org/citation.cfm?id=1769477

>
> Hope this helps
> Kwangsu
>
> On Oct 23, 2010, at 6:51 AM, Ilya Goldin wrote:
>
> > Yes, Ed, that's what I mean, reciprocity in the context of e.g.,  
> > authors telling reviewers whether the feedback was helpful.
>
> > Any pointers would be appreciated.
>
> > Thanks,
> > Ilya
>
> > ----- Original message -----
> >> Hi Ilya,
>
> >> What do you mean by "back-evaluations"?   I think you are saying  
> >> that the
> >> authors rate the reviewers who rated them.   While I don't know of  
> >> any
> >> research myself, I can ask this question on the POD list, which  
> >> includes
> >> many faculty developers interested in various kinds of  
> >> assessment.   If
> >> that is what you mean, my practice has been that a third party  
> >> evaluates
> >> the reviewers ... which seems to evade the issue.   Thanks, Ed
>

> >>>>> Ilya Goldin <ilyagol...@gmail.com> 10/22/2010 2:01 PM >>>

[Kwangsu Cho]

Ilya:

My immediate answer is yes to your question
although the answer is from my vague memory ;)

Fyi, I have a few more papers on peer review that are recently
published.

- Cho, K. & MacArthur, C. (in press). Learning by reviewing. Journal
of Educational Psychology.
- Cho, Y. & Cho, K. (2010). Peer reviewers learning from giving
comments. Instructional Science.
http://www.springerlink.com/content/n6413858825h06t0/
- Cho, K., & MacArthur, C. (2010). Student revision with peer and
expert reviewing. Learning and Instruction, 20, 328-338.
This is a part of Special Issue on peer assessment.
http://www.sciencedirect.com/science/issue/6021-2010-999799995-1824697

Hope they help.
Kwangsu