Utopia II

[Brian Nosek]

This group was highly reinforcing the last time that I circulated a
manuscript.  We received lots of great feedback rapidly.  I am
attaching "part 2" of the Scientific Utopia papers.  Like Roger's,
this is intended for a special issue of Perspectives on Psychological
Science concerning replication.  This one is essentially a missive of
how we got started on the Open Science Framework.

We welcome any and all feedback - privately or publicly.  If you make
line editing suggestions in the manuscript itself, please leave
tracked changes turned on.

---------------------------
Scientific Utopia: II. Restructuring incentives and practices to
promote truth over publishability

An academic scientist’s professional success depends on publishing.
Publishing norms emphasize novel, positive results.  As such,
disciplinary incentives encourage design, analysis, and reporting
decisions that elicit positive results and ignore negative results.
Prior reports demonstrate how these can inflate the rate of false
effects in published science.  Further, because of incentives for
novelty over replication, published false results then persist in the
literature unchallenged.  This reduces efficiency in knowledge
accumulation.  Previous suggestions to address the problem are not
likely to be effective.  For example, journals for negative results
and raising publishing standards will not work because they do not
realign incentives.  The persistence of false results can be reduced
with strategies that make the ultimate, abstract motivation – getting
it right – competitive with the shorter-term, concrete incentive –
getting it published.  We develop strategies for improving scientific
practices that account for ordinary human motivations and self-serving
biases.

[Michael C. Frank]

Hi all,

In the spirit of the manuscripts that have been circulating recently (which I've really enjoyed reading), I thought I'd send another around.

As I mentioned a couple of months ago, Rebecca Saxe and I have been working on a manuscript on using experimental methods courses to teach the practice of replication. I've attached our most recent version here (title and abstract below). We would very much appreciate any feedback from anyone in the group!

thanks,

Mike

---
Teaching replication to promote a culture of reliable science

Statistical and methodological concerns have led to doubts about the reliability of the experimental psychology literature. Replication is held as the gold standard for ensuring reliability. But conducting exact replications is expensive, time-consuming, and unrewarded under current publication practices. So who will do the replications? Our answer is that students in laboratory classes should replicate recent findings as part of their training in experimental methods. In our own courses, we have found that replicating cutting-edge results is exciting and fun, it gives students the opportunity to make real scientific contributions (provided supervision is appropriate), and it provides object lessons about the scientific process, the importance of reporting standards, and the value of openness.

[Joshua Foster]

I love this paper! Do either of you have a course syllabus, outline, or something else that you could share that lays out specifically how you go about each stage of the class project? For example, how do your students select projects? It sounds like in the undergraduate version they are given a list of possibilities, but do they then read each article (or each article that they are interested in) and do a feasibility analysis? I've considered doing something like this, but I've never felt confident that I could pull it off. I'm worried that I'm going to get a few weeks into the projects and then experience a catastrophic problem that I failed to anticipate. Having a manual, or just seeing how someone else does this, would make me more confident. Anyway, great article!

______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548
http://www.southalabama.edu/psychology/Faculty_Foster.htm

>>> "Michael C. Frank" <mcf...@stanford.edu> 5/24/2012 10:08 AM >>>

[Bryan Burnham]

Dear Dr. Frank,

I agree with Joshua; this is a wonderful manuscript you've sent. Is there any way I could get a course syllabus or something similar? I am definitely going to consider incorporating this replication idea into my methods courses next year.

Cheers!

Bryan Burnham

--
Bryan R. Burnham, PhD
Assistant Professor of Psychology
University of Scranton
Scranton, PA 18510

office: 570-941-6687

[Brian Nosek]

Mike and I have discussed the possibility of adapting the replication procedure for the Reproducibility Project into a manual for individuals or classes to use along the lines of Mike's effort.  If anyone is interested in contributing to that, perhaps we could form a small group to get it done (between the RP documentation and Mike/Rebecca's experiences, a good deal of the work is already finished!).  

I think it could itself become a nice publication from the Open Science Collaboration that dovetails on Rebecca and Mike's piece providing the conceptual rationale.

[Though if Mike already has something put together, all the better.] 

[Jeffrey Spies]

This would also be a nice use-case of the OSF; I'm in.

J.

[Joshua Foster]

Sounds like a great idea. I'd love to help.

______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548
http://www.southalabama.edu/psychology/Faculty_Foster.htm

>>> Jeffrey Spies <jsp...@gmail.com> 5/24/2012 12:47 PM >>>

[Michael C. Frank]

Thanks, Brian and Joshua for the positive responses! We will talk things over and figure out what would be appropriate/useful to circulate. Perhaps some more details in the paper, or even a web link for syllabi, would be useful.

much appreciated!

Mike

[sheila miguez]

Would it be possible to get lectures from this on one of the online
courseware platforms? I've been hoping to see a methods class go
online.

--
sheila

[Rebecca Saxe]

Thanks so much, Josh!

I do have the syllabus for the undergraduate class I teach. Will be happy to forward it ASAP.

[sheila miguez]

Hi! Please share it online for lurkers. I am enjoying the discussion.

On Thu, May 24, 2012 at 1:41 PM, Rebecca Saxe <sa...@mit.edu> wrote:
> Thanks so much, Josh!
>
> I do have the syllabus for the undergraduate class I teach. Will be happy to forward it ASAP.

--
sheila

[Michael C. Frank]

Brian, Joshua, Jeffrey,

Great, I'm glad there is interest in this. If others are interested, let me know and we can coordinate off-list.

best,

Mike

[Denny Borsboom]

Great work, thanks for sharing this. I'll share this with the educational monster and see what it thinks.

Best

Denny

--
Denny Borsboom
Department of Psychology
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands
+31 20 525 6882
d.bor...@uva.nl
http://sites.google.com/site/borsboomdenny/dennyborsboom

[Brian Nosek]

Thanks much for the helpful comments on part II of Scientific Utopia.
The revised version is attached for your viewing pleasure or rapid
recycling.