Reconsidering the publication process more generally

[Jody Culham]

In addition to the discussions about open access, it may also be worth
discussing whether the traditional peer review structure is broken and
replaceable with newer models that take advantage of Web 2.0
technology.

There is a provocative article on this:
Kravitz, D. J. & Baker, C. I. (2011). Toward a new model of
scientific publishing: discussion and a proposal. Frontiers in
Computational Neuroscience.

Open access link:
http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2011.00055/abstract

Here are some stats from the article that I found interesting (Table
1). They're from cognitive neuroscience but I think the results would
be similar in perception and attention research:
33% of manuscripts are accepted in the first journal to which they are
submitted; On average papers require submission to 2.1 journals
(range: 1-6), with 2.6 revisions (range: 1-6), requiring 6.3 reviews
(range: 2-15), and taking 221 days (range: 31-533) from first
submission to publication. By the authors' calculations, this adds an
indirect peer review cost (prorated by a postdoc's salary for an
estimate of 6 hours per review) of $840 per paper and an indirect
revision cost covering researcher labour of ~$1600 per paper. The
frequency histograms for citation rates for papers in top-, mid- and
lower-tier journals shows considerable overlap (Fig. 2).

Here's a quote from the paper that I liked (p. 4):
"One study (Peters and Ceci, 1982) resubmitted 12 articles already
published in high-tier journals with different authors names and
institutions. First, only three of the papers were detected as already
published, and at a time when the number of published papers was much
lower than it is today. Second, eight of the nine remaining papers
were rejected, none for novelty, but generally for “serious
methodological flaws.” This result might suggest a systematic bias by
Reviewers or that peer review itself is unreliable."
and a link to the cited paper: http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=6577844

[Alex Holcombe]

Too right, Jody- those numbers show how wasteful the current system
is. There has to be a better way, and I'm sympathetic to the Kravitz &
Baker proposal as well as some others. And what evidence we have finds
that reviewers' verdicts are correlated only very weakly (there's a
PLoS ONE article somewhere I read about this).

We might consider asking existing journals to try out some of the
features in the proposed systems. People at the open-access journals
might be more sympathetic, and find it easier to implement things
without running afoul of publisher copyright constraints. In the case
of the closed-access journals, if the community abandons one or more
of them, they could be replaced with one of these proposed web-
centered systems.

It seems we ought to formulate some options to mention at VSS, as the
prospect of these new ideas might get people excited about a positive
change, not just a rejection of the most offensive elements of the old
system.

[Lee de-Wit]

Author owned copyright, in open access journals, with improved models
of peer review, is certainly needed. Nikolaus Kriegeskorte, has also
outlined an alternative to the current peer review model
http://futureofscipub.wordpress.com/

Publications (and their peer review) are however just one way in which
we should think about science shifting to a more open model. The
Internet provides much more direct means of sharing data, ideas and
programming code which could massively facilitate the pace of
scientific research.

Michael Nielsen discusses in much broader terms the many ways in which
research could be improved by shifting to a genuine model of ‘open
science’. This Ted talk is a good introduction to this.
http://www.ted.com/talks/michael_nielsen_open_science_now.html

Particularly on the issue of sharing data, Jelte Wicherts has a number
of publications highlighting that sharing data would not just be nice,
but may be truly critical to maintaining standards in psychology.
Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case
Nature, 480, 7
Wicherts, J. M., Bakker, M. & Molenaar, D. (2011). Willingness to
share research data is related to the strength of the evidence and the
quality of reporting of statistical results. PLoS ONE