Re: [OPENCAFE-L] The 'one shot' scholarly communication talk
Rick Anderson
That’s certainly been my experience as both an author and a reviewer. I’ve reviewed countless articles, and can think of perhaps one that I returned with no suggestions and a recommendation that it be published as is. Virtually every other one has had substantive problems that I was able to point out (and probably others that I missed and another reviewer caught). When those articles were eventually published, they were markedly better than when they were submitted. Of course, I’ve experienced the same thing as the author of peer-reviewed articles.
So when I hear that studies have suggested that peer review does little or no good, it’s puzzling in light of my personal experience. That’s not enough to lead me to reject such assertions out of hand -- I recognize that my personal experience constitutes a relatively small data set -- but it puzzles me.
From:
Anthony Watkinson <anthony....@btinternet.com>
Date: Thursday, February 8, 2024 at 8:20 AM
To: Glenn Hampson <gham...@nationalscience.org>, Rick Anderson <rick_a...@byu.edu>, "'OPENC...@LISTSERV.BYU.EDU'" <OPENC...@LISTSERV.BYU.EDU>
Cc: "'osi20...@googlegroups.com'" <osi20...@googlegroups.com>
Subject: Re: RE: [OPENCAFE-L] The 'one shot' scholarly communication talk
Hi Glenn, Rick and others.
The strange thing is that when one does peer review the impression is for both reviewer and reviewed that this is a worthwhile enterprise leading to a better result - at least that is my experience.
Anthony
------ Original Message ------
From: gham...@nationalscience.org
To: rick_a...@byu.edu; OPENC...@LISTSERV.BYU.EDU Cc: osi20...@googlegroups.com
Sent: Thursday, February 8th 2024, 15:52
Subject: RE: [OPENCAFE-L] The 'one shot' scholarly communication talk
Gladly Rick. Here are the citations from my 2020 BRISPE presentation (some studies, some articles). There are many others before and since---this is just a sample:
· Kelly, J, T Sadeghieh, K Adeli. 2014. Peer Review in Scientific Publications: Benefits, Critiques, & A Survival Guide. EJIFCC. 2014 Oct 24;25(3):227-43. PMID: 27683470; PMCID: PMC4975196.
· Willis, Michael. 2020. Peer review quality in the era of COVID. https://www.wiley.com/en-us/network/publishing/research-publishing/trending-stories/peer-review-quality-in-the-era-of-covid-19
· Smith, Richard. 2006. Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine vol. 99,4 (2006): 178-82. doi:10.1258/jrsm.99.4.178
· Horbach, SPJM, and W Halffman. 2019. The ability of different peer review procedures to flag problematic publications. Scientometrics 118, 3 39–373
· Tennant, JP, and T Ross-Hellauer. 2020. The limitations to our understanding of peer review. Res Integr Peer Rev 5, 6. Doi: 10.1186/s41073-020-00092-1
· Open Scholarship Initiative. 2016. Report from the OSI2016 Peer Review Workgroup. doi: 10.13021/G8K88P
o “Peer review is the worst form of evaluation except all those other forms that have been tried from time to time.-with apologies to Winston Churchill”
From: Rick Anderson <rick_a...@byu.edu>
Sent: Thursday, February 8, 2024 7:29 AM
To: Glenn Hampson <gham...@nationalscience.org>; OPENC...@LISTSERV.BYU.EDU
Cc: osi20...@googlegroups.com
Subject: Re: [OPENCAFE-L] The 'one shot' scholarly communication talk
> but many studies have complained over the years that the evidence is unclear whether peer
> review actually improves research (beyond making articles more readable).
Glenn, could you share links to a few of these?
From: <osi20...@googlegroups.com> on behalf of Glenn Hampson <gham...@nationalscience.org>
Date: Thursday, February 8, 2024 at 7:27 AM
To: "OPENC...@LISTSERV.BYU.EDU" <OPENC...@LISTSERV.BYU.EDU>
Cc: "osi20...@googlegroups.com" <osi20...@googlegroups.com>
Subject: RE: [OPENCAFE-L] The 'one shot' scholarly communication talk
Wow. Living on the West coast of the US can be rough. By the time your day gets started, listserv conversations can be almost over! If I may, there are a couple of issues here that I see differently than my esteemed colleagues.
First, to this whole notion introduced by my friend Ricks and Lisa that peer review is highly effective at weeding out garbage and allowing good scholarship to get published: This is certainly true for the editorial process in general (like the desk rejection process), but it isn’t true of peer review. The peer review process is highly regarded by researchers, seen as a signal of quality (see https://bit.ly/3otwKRs), and highly valued by funders and institutions, but many studies have complained over the years that the evidence is unclear whether peer review actually improves research (beyond making articles more readable).
This process also varies by journal (see note below) and is highly subject to bias, as Daniel mentions---by idea, gender, nationality, etc.
Here’s a link to a presentation I gave a few years ago on this topic. There’s too much detail to bore you with in a listserv email but the presentation has references included if you want to dig deeper: BRISPE-presentation-final-Hampson.pdf (osiglobal.org). In particular, I suggest you read Melinda Baldwin’s great paper on the history of peer review (Baldwin, Melinda. 2018. Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States. Isis, volume 109, number 3). The peer review system we use today is essentially a byproduct of US Congressional oversight in the mid-1970s; it took decades thereafter for this process to become widely used throughout the world.
So what do you tell your students in your one-shot, Melissa? I don’t know. Maybe that peer review is quality-control process we invented to help “monitor” science and now it’s kind of an institution unto itself with a mythology larger than its actual value to science?
Regarding Pooja’s story about gatekeeping, I know this might not make your colleague feel better, but most papers are rejected at least once, for any number of reasons (as Jean-Claude explained, like a bad fit with the journal’s focus). Across all kinds of journals, the average rejection rate of articles is a whopping 60-65% (https://doi.org/10.3145/epi.2019.jul.07). Individual rates vary widely by journal, ranging from 0-90% and higher. About 20% of papers get rejected before peer review for being out of scope, among other reasons (see https://bit.ly/2YnYoVv). All this said, most papers eventually get published somewhere. Two-thirds of preprints posted before 2017 were later published in peer-reviewed journals within 12-18 months (see https://doi.org/10.7554/eLife.45133). Also, if your colleague is submitting to 65 different journals, it seems like they might be casting a net that is too wide (and too unfocused), which might not be the best approach.
And finally, to Toby and Danny’s big-picture thinking, here’s an infographic OSI created a few years ago to show how review and publishing fit into (and feed into) the full idea lifecycle: OSI-Infographic-1.0: The Idea Lifecylce (osiglobal.org). There’s a lot more to research than just publishing, obviously, but to Jean-Claude’s point, publishing still plays a critical role (and it always has throughout the history of science). What form this takes in the future is where so much of the attention and effort in the OA reform space has been directed.
Best regards,
Glenn
Glenn Hampson
Executive Director
Science Communication Institute (SCI)
Program Director
Open Scholarship Initiative (OSI)
Note: Generally speaking, specialty and prestige journals provide high quality peer review; even some preprint servers are experimenting with new forms of peer review. Regional journals don’t always provide the kind of peer review required by specialty journals; peer review quality here varies widely.
From: OpenCafe-l <OPENC...@LISTSERV.BYU.EDU> On Behalf Of Daniel Kulp
Sent: Thursday, February 8, 2024 6:32 AM
To: OPENC...@LISTSERV.BYU.EDU
Subject: Re: [OPENCAFE-L] The 'one shot' scholarly communication talk
At the end of the day, peer review is run by people (editors, reviewers, etc.), and people are susceptible to bias. Is peer review perfect? No, it’s not. It is likely the best we have, at the moment. I certainly support experiments in the publishing industry, but I have yet to see a process which is consistently better and able to applied at scale. That would be how I would frame peer review to students.
Daniel Kulp, PhD
Founder, PIE Consulting
Publicationintegrity.com
On Feb 8, 2024, at 5:21 AM, Jean-Claude Guédon <jean.clau...@UMONTREAL.CA> wrote:
My own take on peer review is that it is part of a larger process which I have sometimes described as the "Great Conversation" that stands behind knowledge producing. Voltaire says somewhere - and I am paraphrasing - it is difficult to live without certainty, but believing that certainty exists is ridiculous. Knowledge production works exactly at that level, and that is where it differs from believing or convictions. Peer review is part of the process one can use to allow the best forms of human thinking to percolate to the surface and become reference points for further evolution of knowledge. Knowledge can only claim reliability, not certainty.
Rick is right when he says that, when executed competently and honestly, peer review is highly effective. The main problem is that parts of the process can remain quite opaque. For example, that "desk rejection" Rick mentions generally involves one person only. That person - the editor - may have two divergent objectives in his/her mind: on the one hand, his/her notion of quality, and, on the other, the effect of the article on the general standing of the journal, especially in a tightly controlled competition system such as the impact-factor driven mechanism.
Imagine yourself in the following situation: you have room (i.e. resources) to publish one article. You have two submissions. One article is on a hot topic but its quality is ho-hum. The other one appears stellar but on a topic that is more marginal in the present development of knowledge (perhaps it is not yet well understood, or whatever). Which principle will be used at the desk rejection level? The first article is bound to improve your impact factor; the second article may lower your impact factor. This is because the relationship of the impact factor to quality is both tenuous and ambiguous.
More generally, how is peer review affected by the fact that scientific articles and journals are two different kinds of objects but have been entangled with each other since the advent of print. And this leads to another question: in a digital world, do journals still matter, and, if so, how can they matter? How do journals relate to platforms? To communities? Etc.
Jean-Claude Guédon
On 2024-02-07 23:23, Rick Anderson wrote:
Here’s how I explain peer review to students:
When an author submits her paper to a peer-reviewed journal, the journal’s editor gives it a first look. If it doesn’t appear to be up to scratch (which can mean any number of things: obviously poor methodology, illegibility, irrelevance, etc.) then the editor rejects it -- we call this “desk rejection.” If it looks like it has promise, the editor sends it out to one or more (usually at least two) reviewers. They’re called “peer” reviewers because they work in the same field as the author, or a closely adjacent one, so they’re in a good position to evaluate the scholarship. The reviewers are asked to read the paper more closely and evaluate it on its scholarly merits: is its methodology sound; do the conclusions proceed from the data; is it well organized and cogently written; do the cited works actually support the arguments in support of which they’re cited; etc. The reviewers submit reviews with recommendations as to whether the article should be rejected, or returned for revision, or published as is. This process may involve two or three rounds before the paper is finally published or rejected.
It's by no means a fail-safe system, but when executed competently and honestly, it’s highly effective at weeding out garbage and allowing good scholarship to get published. Unfortunately, the competence and honesty of journals is highly variable. Sometimes they get into bed with corporations who want to see certain things published; sometimes editors of different journals collude with each other to require authors to cite each other’s publications; and in recent years, there’s been a growing industry of journals that dishonestly claim to carry out peer review when in fact they will publish anything submitted to them as long as the author pays a publication fee. So before submitting to a journal, it’s really important to do your due diligence.
From: OpenCafe-l <OPENC...@LISTSERV.BYU.EDU> on behalf of Danny Kingsley <da...@DANNYKINGSLEY.COM>
Reply-To: Danny Kingsley <da...@DANNYKINGSLEY.COM>
Date: Wednesday, February 7, 2024 at 8:03 PM
To: "OPENC...@LISTSERV.BYU.EDU" <OPENC...@LISTSERV.BYU.EDU>
Subject: [OPENCAFE-L] The 'one shot' scholarly communication talk
Hi everyone,
I’m picking up in a new thread something Melissa noted:
As a librarian, I need to be able to stand in front of a class of freshmen, as I am about to do tonight, to explain what peer-review is and why it's the gold standard for what they cite in their papers, and to be able to say it with a straight face without feeling like a liar. For those of you who know what a "one-shot" is, you know we do NOT have time to explain the intricacies of the scholarly publishing industry, its good and bad financial incentives, etc., even if we understand them fully ourselves. We don't even have time to explain all that to graduate students.
This is a really good point for discussion.
How do people approach this type of explanation? I am thinking there is a parallel with the difference between what is written in textbooks and what is happening in the scholarly literature. Textbooks tend to present information as ‘decided’, information published in the literature is the ongoing debate. Textbooks change perspective and ideas slowly, a paper can get shot down in weeks/months.
So, do we provide the ‘textbook' version to students: “This is how science works, a research team find something out, write it up, send it to a journal, it gets sent to experts in the field, they comment, amendments are made and then it is published”.
Or do we bring in some of the broader picture: “Researchers don’t get paid to publish. Publication is the way researchers get ‘prestige’ - the better their paper and (more commonly) the place they publish it in ‘counts’ towards their academic standing. There are systems that count how many papers people have published, where they have published and how many other people have subsequently cited their work. These numbers are fed into most decision making in research - whether someone gets a promotion, whether they get a grant, how an institution fares in national ‘research excellence’ exercises and how universities get ranked."
Or do we lay it down: “The very narrow focus on what constitutes 'success’ in research has unfortunately resulted in some very poor behaviour…..”
I am conscious that when this is new to people it can seem overwhelming. A comment at last year’s AIMOS conference (which consisted of multiple presentations about research on research, uncovering a swathe of issues) was that it was very depressing and it meant it was hard to believe anything that was published. To be honest when you read articles like these https://www.theguardian.com/science/2024/feb/03/the-situation-has-become-appalling-fake-scientific-papers-push-research-credibility-to-crisis-point (which is referring to activity all over world) you can get depressed.
My response is that it is good we are lifting the lid on this - these are the steps we make towards fixing the problems.
But we want our community to ‘be alert not alarmed’.
How do people approach this discussion in their own institutions?
Danny
Dr Danny Kingsley
Scholarly Communication Consultant
Visiting Fellow, Australian National Centre for the Public Awareness of Science, ANUAdjunct Senior Lecturer, Charles Sturt University
Member, Board of Directors, FORCE11
Member, Australian Academy of Science National Committee for Data in Science
---------------------------------------
e: da...@dannykingsley.com
m: +61 (0)480 115 937
t:@dannykay68b: @dannykay68.bsky.social
o: 0000-0002-3636-5939
Access the OPENCAFE-L Home Page and Archives
To unsubscribe from the OPENCAFE-L send an email to: OPENCAFE-L-si...@LISTSERV.BYU.EDU
Access the OPENCAFE-L Home Page and Archives
To unsubscribe from the OPENCAFE-L send an email to: OPENCAFE-L-si...@LISTSERV.BYU.EDU
Access the OPENCAFE-L Home Page and Archives
To unsubscribe from the OPENCAFE-L send an email to: OPENCAFE-L-si...@LISTSERV.BYU.EDU
Access the OPENCAFE-L Home Page and Archives
To unsubscribe from the OPENCAFE-L send an email to: OPENCAFE-L-si...@LISTSERV.BYU.EDU--
As a public and publicly-funded effort, the conversations on this list can be viewed by the public and are archived. To read this group's complete listserv policy (including disclaimer and reuse information), please visit http://osinitiative.org/osi-listservs.
---
You received this message because you are subscribed to the Google Groups "The Open Scholarship Initiative" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unsub...@googlegroups.com">osi2016-25+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osi2016-25/DM4PR17MB60640E4C6121008C690CBB07C5442%40DM4PR17MB6064.namprd17.prod.outlook.com.
--
As a public and publicly-funded effort, the conversations on this list can be viewed by the public and are archived. To read this group's complete listserv policy (including disclaimer and reuse information), please visit http://osinitiative.org/osi-listservs.
---
You received this message because you are subscribed to the Google Groups "The Open Scholarship Initiative" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unsub...@googlegroups.com">osi2016-25+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osi2016-25/DM4PR17MB60648FC9FC326C8D453F1D91C5442%40DM4PR17MB6064.namprd17.prod.outlook.com.
David Wojick
<image001.jpg>
To view this discussion on the web visit https://groups.google.com/d/msgid/osi2016-25/2C8D8A5C-D44F-4593-B65C-959B9B17D548%40byu.edu.