Paper Draft -- Scientific Utopia: Opening Scientific Communication

[Brian Nosek]

Attached is a monkey that is now coming off of my back. This is an
article for Psychological Inquiry - a journal that has one target
article and then many commentaries. The goal is to facilitate debate
on the topic. The due date was April 1, just missed it (geological
time).

This is a draft version. The editor will give some feedback very
soon, and then we will have a short time frame to edit and finalize
before the paper goes out to the commenters for their review. We have
waded into many topics with this manuscript, and still only scratched
the surface. Also, the reality is changing very rapidly right now, so
it is a delicate balance to make this relevant in journal-time when
much of the action is happening in real time. [You'll recognize the
irony in reading the manuscript.]

So, if any of you have time and inclination to read soon and offer
feedback, we will be delighted to get it. We want this to be as
compelling as possible and, hopefully, stimulate good debate. We
might have as little as a couple of weeks for revisions (I don't know
for sure yet). But, not to worry if you don't have time - this has an
unusual time crunch because of my own delays.

Feedback can be sent publicly or privately, up to you.

Thanks,

Brian

[Brian Nosek]

I hear from the editor that she'd like to get this out to the
commenters by early next week. So, if you happen to read it and feel
inclined to send feedback, we will probably be able to incorporate
things that we receive by this weekend. If you happen to read it
later and wish to send feedback, we will be delighted to get it and
then feel regret about how we can't improve the paper in light of it.
[But we do get to respond to commentaries, so I'd be able to make
clarification points and raise alternatives there that were not
considered in the original ms.]

Apologies for a ridiculously compressed timeline, it is my own fault.

[Roger Giner-Sorolla]

This is a great Inquiry target. That isn't to say that I couldn't think of counter-arguments, but hey, that is what Inquiry is for, and you seem to anticipate most of them. In a sense, a foretaste of the open review you presage.

I'd actually like to cite this in the paper I'm writing for the PoPS special issue because it would help me streamline what I have to say, which is that at some point in this procedure we can start to separate evaluation of methodological rigor and interpretation from evaluation of the strength or consistency of results as a criterion for sharing research. In effect, this will be facilitated by the focal selection you hint at, where we want to see everything no matter how messy in our specific field (filtered by rigor, or evaluated by ourselves for same) but only have time for well-supported conclusions in peripheral fields. The difference is that we can be sure that well-supported conclusions have gone through a more systematic review process in their own sub-arena, in a system where there are incentives to share the file drawer and even publish your own.

[Brian Nosek]

We receive a lot of feedback quickly. Thanks, it was extremely
helpful. I am attaching the final version, FYI.

Feel free to circulate. If you are itching to write a commentary, you
could email the editor, Ronnie Janoff-Bulman and inquire if there will
be space: jan...@psych.umass.edu

[Ravi Iyer]

Very cool, Brian & Yoav and many thanks!

I think one dimension that your argument is likely to rest upon is
whether or not crowdsourcing actually can produce better science than
a more closed process. To that end, I'd invite those interested to
consider the elements of crowdsourcing that lead to better (or worse)
results than a panel of experts, which I think is well laid out in
this paper.

http://faculty.fuqua.duke.edu/~jsoll/Larrick%20Mannes%20Soll%20WOC%20chapter%20Final.pdf

Note that this is a topic that is of interest to not just
psychologists, but everyone who wants to answer questions in the
world. Almost every technology company out there is trying to figure
out how to crowdsource more effectively.

Ravi

[Roger Giner-Sorolla]

So what is Scientific Utopia II going to be about?

[Brian Nosek]

We are going through references today and should be circulating tonight or tomorrow.  

It is for the same PPS special issue as yours and looks like it will be quite complementary.  We give about 5 words to perfect data, but discuss the same issue.

Title + Abstract

Scientific Utopia: II. Restructuring incentives and practices to promote truth over publishability

Professional success for scientists depends on publishing.  Publishing norms emphasize novel, positive results.  As such, incentives for professional success encourage design, analysis, and reporting decisions that elicit positive results, and ignore negative results. Prior reports demonstrate how these can inflate the false positive rate in published science (e.g., Greenwald, 1975; Simmons, Nelson, & Simonsohn, 2011).  False results can then persist in the literature because of the incentives for novelty over replication, and the unavailability of original data and materials.  While the problem is widely understood, little progress has been made to address it.  We explain why, and provide suggestions for effective change.  The persistence of false results can be reduced by increasing transparency of the entire scientific process, and by capitalizing on the ultimate motivation - getting it right - over the shorter-term incentive - getting it published. 

[Roger Giner-Sorolla]

That sounds great. Looking forward to calibrating and cross-referencing our ideas.