RE: [scisip] Journal Prestige and Reliability

[Glenn Hampson]

Hi Steve,

 

Do you agree with the author’s conclusion (“there is no evidence that articles published in higher ranking journals are methodologically stronger”)? To me anyway, it isn’t supportable for two main reasons:

 

1. His study is based on impact factors, which are a notoriously inept measure of who knows what exactly ---it’s not hard to find articles calling for the death of impact factors. This one is my favorite: https://www.peerageofscience.org/a-tale-of-two-distributions/. Stats nerds will appreciate this take. And,
2. It generalizes from very specific examples in specific fields of study to broad conclusions about all journals from all fields of study. That is, the quality of computer models in papers about molecular structures are better in lower impact factor journals than higher, and the statistical power in psych studies is better in lower than higher, and experimental design is better in animal studies, etc. (assuming, of course, that the sampled journals didn’t lie about their impact factors, which is a problem seen with predatory journals). Ergo “(1) experiments reported in high-ranking journals are no more methodologically sound than those published in other journals; and (2) experiments reported in high-ranking journals are often less methodologically sound than those published in other journals.” I don’t see how this generalization is warranted, but then maybe I’m reading this all wrong.

 

So I don’t know---many don’t go publishing in the “Journal Nobody Reads” just yet?

 

Best,

 

Glenn

 

 

Glenn Hampson
Executive Director
Science Communication Institute (SCI)
Program Director
Open Scholarship Initiative (OSI)

2320 N 137th Street | Seattle, WA 98133
(206) 417-3607 | gham...@nationalscience.org | nationalscience.org

 

 

 

From: Science of Science Policy Listserv [mailto:SCI...@LISTSERV.NSF.GOV] On Behalf Of Fiore, Steve
Sent: Tuesday, February 20, 2018 7:07 PM
To: SCI...@LISTSERV.NSF.GOV
Subject: [scisip] Journal Prestige and Reliability

 

Hi Everyone - There is a new article out summarizing various findings that examined journal prestige in relation to the quality of the research published in the journals.  The article is fully open access and can be found at this link (https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full).  But I've also pasted a table that summarizes the various disciplines, the criteria for research quality/reliability, and what was found in association with journal ranking.

 

Best,

Steve

 

--------

Stephen M. Fiore, Ph.D.

Professor, Cognitive Sciences, Department of Philosophy (philosophy.cah.ucf.edu/staff.php?id=134)

Director, Cognitive Sciences Laboratory, Institute for Simulation & Training (http://csl.ist.ucf.edu/)

University of Central Florida

sfi...@ist.ucf.edu

 

 

Prestigious Science Journals Struggle to Reach Even Average Reliability
Björn Brembs, Universität Regensburg, Regensburg, Germany

In which journal a scientist publishes is considered one of the most crucial factors determining their career. The underlying common assumption is that only the best scientists manage to publish in a highly selective tier of the most prestigious journals. However, data from several lines of evidence suggest that the methodological quality of scientific experiments does not increase with increasing rank of the journal. On the contrary, an accumulating body of evidence suggests the inverse: methodological quality and, consequently, reliability of published research works in several fields may be decreasing with increasing journal rank. The data supporting these conclusions circumvent confounding factors such as increased readership and scrutiny for these journals, focusing instead on quantifiable indicators of methodological soundness in the published literature, relying on, in part, semi-automated data extraction from often thousands of publications at a time. With the accumulating evidence over the last decade grew the realization that the very existence of scholarly journals, due to their inherent hierarchy, constitutes one of the major threats to publicly funded science: hiring, promoting and funding scientists who publish unreliable science eventually erodes public trust in science.

 

 

Table 1. Overview of the cited literature on journal rank and methodological soundness.

 

 

########################################################################

To send to the list, address your message to: SCI...@listserv.nsf.gov

To subscribe to the list: send the text “subscribe SCISIP” to list...@listserv.nsf.gov

To unsubscribe: sent the text “unsubscribe SCISIP” to list...@listserv.nsf.gov