Fwd: The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications
Jodi Schneider
From: Stephen Gallo <sga...@aibs.org>
Date: Mon, Oct 31, 2016 at 10:08 AM
Subject: The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications
To: SCIT...@list.nih.gov
Hi All,
Here is a link to our latest article about reviewer expertise and scoring relative to the evaluation of grant proposals. I apologize for the cross-posting.
ABSTRACT
Although the scientific peer review process is crucial to distributing research investments, little has been reported about the decision-making processes used by reviewers. One key attribute likely to be important for decision-making is reviewer expertise. Recent data from an experimental blinded review utilizing a direct measure of expertise has found that closer intellectual distances between applicant and reviewer lead to harsher evaluations, possibly suggesting that information is differentially sampled across subject-matter expertise levels and across information type (e.g. strengths or weaknesses). However, social and professional networks have been suggested to play a role in reviewer scoring. In an effort to test whether this result can be replicated in a real-world unblinded study utilizing self-assessed reviewer expertise, we conducted a retrospective multi-level regression analysis of 1,450 individual unblinded evaluations of 725 biomedical research funding applications by 1,044 reviewers. Despite the large variability in the scoring data, the results are largely confirmatory of work from blinded reviews, by which a linear relationship between reviewer expertise and their evaluations was observed—reviewers with higher levels of self-assessed expertise tended to be harsher in their evaluations. However, we also found that reviewer and applicant seniority could influence this relationship, suggesting social networks could have subtle influences on reviewer scoring. Overall, these results highlight the need to explore how reviewers utilize their expertise to gather and weight information from the application in making their evaluations.
Thanks
Steve
Technical Operations Manager
American Institute of Biological Sciences Scientific Peer Advisory and Review Services
Fax:703-674-2509
Email: sga...@aibs.org
Please follow us on Twitter! @AIBS_SPARS
This message is confidential and should only be read by its intended recipients. If you have received it in error, please notify the sender and delete all copies.
The Science of Team Science (SciTS) listserv facilitates knowledge sharing among individuals engaged in, studying, facilitating, and supporting team science, in the US and internationally. It is maintained by the SciTS Team of the National Cancer Institute of the U.S. National Institutes of Health.
https://www.teamsciencetoolkit.cancer.gov/Public/ToolkitTeam.aspx
VISIT THE TEAM SCIENCE TOOLKIT, a one-stop-shop for resources to help you lead, manage, facilitate, support, or study team-based research - www.teamsciencetoolkit.cancer.gov
TO SUBSCRIBE:
Send an email with a blank subject line to: list...@list.nih.gov. The message body should read: subscribe SciTSlist [your full name]. Please do not include the brackets. For example, for Robin Smith to subscribe, the message would read: subscribe SciTSlist Robin Smith. You will receive a confirmation email.
TO UNSUBSCRIBE:
Send an email with a blank subject line to: list...@list.nih.gov. The message body should read: SIGNOFF SCITSLIST.
TO POST TO THE LISTSERV:
Send an email to SciT...@list.nih.gov. Any subscriber may post to the list.
TO VIEW THE ARCHIVES:
To view the archives of all previous postings, go to: http://list.nih.gov/archives/SciTSlist.html
TO RECEIVE MESSAGES IN A DAILY DIGEST:
The default setting sends you each message as it is posted to the listserv. To receive one daily digest, instead, go to:
http://list.nih.gov/cgi-bin/wa.exe?SUBED1=SciTSlist&A=1 and select “digest” as your subscription type.
TECHNICAL PROBLEMS WITH YOUR SUBSCRIPTION?
Contact the list administrator, Judy Kuan, at: ku...@mail.nih.gov. Please be sure to state that your email is in reference to the SciTS listserv.
******************************************************************************