Incentives: reviewer ranking/awards

[Eugene Agichtein]

ICDE 2010 is experimenting with an interesting reviewer ranking/awards
scheme where the authors rate the "helpfulness" of the reviews, which
is then normalized by dividing by the reviewer score (of course,
authors like positive reviews more :-) ).

The details of the scheme are here: http://www.icde2010.org/index.php/reviewawards

Similar scheme could be used to rank SIGIR reviewers, and maintain
such rankings from year to year (instead of focusing on the extreme
bad cases by collecting SPC complaints).

[Paul McNamee]

A slightly different alternative that I would be in favor of would be
to support (anonymous) ratings by co-reviewers. Somebody who
accumulates sustained 'unhelpful' tags should be counseled or even
disinvited from future reviewing. Somebody who attains repeated
'particularly insightful' ratings should clearly be thanked. Just the
fact that your peer reviewers might rate a weak review of yours, might
motivate extra effort from those who haven't been putting their best
efforts into the process.

I agree keeping data year-to-year would be helpful.

On Mar 29, 11:37 am, Eugene Agichtein <eugene.agicht...@gmail.com>
wrote:

[Gene Golovchinsky]

I think a number of rating schemes are possible, and the details of
any particular scheme are largely unimportant. What is important in my
opinion is the public, persistent recognition of excellence. That is,
we should devise a scheme whereby outstanding reviewers (by whatever
metric or set of metrics we adopt) are listed on the SIGIR or the ACM
web site in a way that they, and others, can refer to. Reviewing is a
largely thankless task, and thus gets lower priority among other tasks
competing for each person's attention. Writing papers is
(occasionally!) rewarded by their acceptance; people who get published
consistently are rewarded with tenure, etc. Since the entire
publication scheme relies on accurate reviewing, we need to make it
easier for people to claim that they are contributing quality
reviewing.

I wrote this up in response to a similar discussion for CHI reviewing.
See <a href="http://palblog.fxpal.com/?p=2392">http://
palblog.fxpal.com/?p=2392</a>. The similarity of the situation
suggests that the policy may be better implemented at the ACM level in
the end, but it may be easier to get individual SIGs to get the ball
rolling.

[Gene Golovchinsky]

I think it's probably important to discount authors' assessments when
the paper is accepted. The goal of the exercise is to identify
reviewers who are effective in communicating how to improve research,
although, of course, the ability to recognize good work should not be
discounted completely.

On Mar 29, 8:37 am, Eugene Agichtein <eugene.agicht...@gmail.com>
wrote:

> ICDE 2010 is experimenting with an interesting reviewer ranking/awards
> scheme where the authors rate the "helpfulness" of the reviews, which
> is then normalized by dividing by the reviewer score (of course,
> authors like positive reviews more :-) ).

--
You received this message because you are subscribed to the Google Groups "SIGIR Meta" group.
To post to this group, send email to sigir...@googlegroups.com.
To unsubscribe from this group, send email to sigir-meta+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sigir-meta?hl=en.