Notable paper: Bargh, Chen, & Burrows (1996) replication attempt

[Brian Nosek]

http://blogs.discovermagazine.com/notrocketscience/2012/01/18/primed-by-expectations-%E2%80%93-why-a-classic-psychology-experiment-isn%E2%80%99t-what-it-seemed/

[Eric-Jan Wagenmakers]

Really great stuff! This should have been published in JPSP.

Cheers,
E.J.

On Sat, Jan 21, 2012 at 9:39 AM, Brian Nosek <no...@virginia.edu> wrote:
> http://blogs.discovermagazine.com/notrocketscience/2012/01/18/primed-by-expectations-%E2%80%93-why-a-classic-psychology-experiment-isn%E2%80%99t-what-it-seemed/
>
>

--
********************************************
Eric-Jan Wagenmakers
Department of Psychological Methods, room 2.16
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands

Web: www.ejwagenmakers.com
Email: EJ.Wage...@gmail.com
Phone: (+31) 20 525 6420

“Man follows only phantoms.”
Pierre-Simon Laplace, last words
********************************************

[Brian Nosek]

PLoS One is the new JPSP!  Prominent articles like these might increase awareness and interest in open access journals among psychologists.  As soon as the field understands these journals as legitimate "good" outlets, there will be a rapid shift in publication norms.  Late, but not too late to the party!

[Bobbie]

Great! But not the first publication of a failure to replicate that
study. See:
http://www.psychfiledrawer.org/chart.php?target_article=1&type=failure

People have been telling me for years about failure to replicate that
study. Can we put a bunch in one place?

And, please... start posting other prior failures (or even prior
unpublished successful replications) there.
(Nah, I don't mean your OSF stuff. I mean previous replication
attempts.)

Bobbie Spellman



On Jan 20, 5:39 pm, Brian Nosek <no...@virginia.edu> wrote:
> http://blogs.discovermagazine.com/notrocketscience/2012/01/18/primed-...
>
>  journal.pone.0029081.pdf
> 254KViewDownload

[Roger Giner-Sorolla]

When I get the time I'm going to put up two (conceptually/procedurally
different) replication studies that a recent editor requested we
remove from a recent paper to leave only the "strongest" one (!!)

[Manuela Barreto]

Hi there
 
I have been following your discussions closely, although unfortunately am unable to participate more at this stage. But I do want to add something to what Roger said in this email below. I completely agree that it is preposterous to pressure authors to throw out conceptual replications from their papers. I just had a desk rejection and the argument was---the first study is just a conceptual replication and the second study is indeed novel but it takes too long to get there. Having said that, I do understand that sometimes there is a need to save journal pages and that until quite recently cutting replications was seen as an easy way to allow for more novel research to be published. Obviously, any of us have now become well aware that replication is not just important but quite crucial to science as a whole and our discipline in particular. Unfortunately, this is not yet the opinion of many journal editors, and this discourages researchers from conducting replications or from including them in their papers. Can something be done about that? Can the attention of editorial boards be directed to such issues? Are we going to wait until that happens, or would some form of action be relevant?
 
Best
Manuela 
 
Manuela Barreto

Professor Social and Organisational Psychology

College of Life and Environmental Sciences

University of Exeter

m.ba...@exeter.ac.uk

(+44-0)1392-722674

http://psychology.exeter.ac.uk/staff/index.php?web_id=Manuela_Barreto

www.manuelabarreto.com

 

 

> Date: Sun, 29 Jan 2012 09:47:23 -0800
> Subject: [OpenScienceFramework] Re: Notable paper: Bargh, Chen, & Burrows (1996) replication attempt
> From: rogerse...@gmail.com
> To: openscienc...@googlegroups.com

[Daniel Lakens]

I think something should and can be done. Journal pages is less of an
issue with online supplemental material. You should at least be
invited to resubmit your article in slightly modified form, moving the
replication to supplementary online material. Obviously, with new open
access online journals (PlosOne, Frontiers, etc) page limitations no
longer exist, and editors should learn to lengthen their attention
span ;)

It is striking that everyone knows that replications are the
cornerstone of empirical science, but researchers don't value them. I
was thinking of writing a Perspectives on Psychological Science
article about this, proposing a 'replication value' - a simple
function that calculates the value of a replication based on the
number of citations and the number of replications. I mentioned this
idea of a 'replication value' earlier here (http://groups.google.com/
group/openscienceframework/msg/680393e03b4c273b) and some people
seemed to like the idea. If anyone is interested to work on a PPS
article (authored by the Open Science Framework Collective, obviously)
on such a thing, let me know.

The replication value could for example be the number of time a study
has been cited devided by (the number of times it is replicated,
squared, plus 1). Journals could have a RV they adorse, depending on
how much they like novel findings, and how much they like less novel
findings. A RV of 50 seems decent, making it worthwhile (for editors)
to publish a direct replication if no others exist, and the paper has
been cited 50 times, and publishing a second replication, when a
finding is cited 200 times, etc.

I know we are all busy, but anyone interested in working on such a
thing, let me know. I think it is much needed, and proposing a
Replication Value might put this issue on the agenda of editorial
boards.

Daniel

Dr. Daniël Lakens
School of Innovation Sciences
Eindhoven University of Technology
Faculty of IE&IS, Room IPO 1.24
Postbus 513, 5600 MB EINDHOVEN
D.La...@tue.nl 040-2472888
http://sites.google.com/site/lakens2/

[Heather Fuchs]

Hi Daniel,

 

I would be interested in working on such a paper.

 

However, I think one thing that should be taken into consideration is the difference between independent replications and replications by the same authors (especially in the form of a multiple study paper). A closer look at all "two study papers" in JPSP from the first half of 2011, for example, shows that a considerably larger proportion (almost 40 percent more according to my preliminary calculations) report hypothesis confirming results in both studies than would be expected in the absence of any publication bias. I don't think that this result is unique to this sample of papers. Thus, I think this trend towards multiple study papers is problematic and one should keep that in mind when constructing any type of replication measure, as anything placing more pressure on authors to produce multiple study papers will likely increase the problem of publication bias.

 

Sincerely,

 

Heather  

 

>>> Daniel Lakens <lak...@gmail.com> 1/30/2012 3:21 >>>

[Daniel Lakens]

Hi Heather,

Cool. Let's see who else is interested, and discuss this further collectively. I would primarily focus on replications by other labs. The number of studies within a single publication is a related matter, but I'm mainly interested in replications of findings that are cited at least a number of times. Those findings are likely to influence scientific theory, and the need for replication to build stronger theories is something that needs to be addressed in social psychological science (the Bargh et al study is a good example of a paper that is cited often, replicated rarely, and which has influenced theories in social psychology).

Daniel

[Brian Nosek]

I was telling Hal Pashler (cc'ed) about this idea a few days ago.  Hal and EJ are putting together a special issue for Perspectives on Psychological Science on issues related to this.  I don't know how much of the special issue is fleshed out yet, but I think Daniel should pitch a piece on replication value to them.  I think it could be done very effectively as a pointed 1-2 page piece laying out the idea and providing some illustrative examples of how it would be used to identify/prioritize replication efforts.

Also, I incorporated the idea into a pitch for journals to do a special issue of replication studies.  As we know, part of the problem of replications is that the incentives are misaligned for valuing replication - for authors, journals, and readers.  Replication value can help adjust those incentives.  Here is a pitch that I sent to an editor for doing a special issue:

--- Special Issue: Replications of high-impact research in SUBDISCIPLINE ---

The "value" of conducting a replication might be roughly indexed by computing the times a study has been cited against the number of times the finding has been replicated (Lakens - in the OSF discussions).  High impact papers with few (or no) published replications have a very high "replication value".  That is, it would really be worth knowing that the effect is reproducible because it is having widespread impact on the field.  

The journal could issue a call for replications that exceed a particular "replication value" threshhold.  People would be asked to submit short summaries of the proposed replication (either that has already been done, or that will be done).  If there are multiple submissions for the same finding, the groups would be asked to produce a joint report.  

The articles could be very short if there was a lot of interest.  It could be an "easy" publication for many folks that have stuff in their file-drawer, and it could produce some very interesting reports.  Further, for replications that have not yet been done, the review process could be completed prior to data collection.  The author(s) write their introduction, methods, and analytic plan for peer review.  If accepted, as long as they follow through on the method as proposed, the paper would published regardless of result.

This realigns incentives: 

authors - "relatively" easy publication if methodology is sound, opportunity to get studies already done out of the file-drawer, opportunity for a relatively highly cited publication on an important topic

journal - because the original papers are high-impact and there are no or few published replications, the replications would likely be cited frequently, also it would be a unique and attention getting strategy on the problems of replication and file-drawer

readers - the uniqueness of the approach could make it an interesting read, especially if there were some standards across papers for reporting style of the replications and some commentary/discussion about the aggregate results and implications; further, because it is only replicating high-impact papers, there would be a lot of interest in what happens.

-------------------------------------

[Kristi Lemm]

I've got a few failed replications of the Bargh walking study as well (two conceptual, one very close to the original).  The first conceptual replication attempt worked, but I was never able to replicate that study again, even after we switched to the stimuli from the original paper.   grr.

-kristi

[Eric-Jan Wagenmakers]

I completely agree that there is something wrong about the incentives,
and I enthusiastically support the special issue idea on replication
of high-impact research! Doing these special issues would fix things
on both sides: credit for running a good replication study and
discredit for publishing research that does not replicate. I think the
latter is also very important. When you think of it, it is rather
strange that someone like Diederik Stapel could fake data for over a
decade -- apparently he did not fear that someone would fail to
replicate his work (did anybody ever try to replicate Stapel?).

On a related note, I think the power of a network such as this one is
that it can act quickly and decisively on high-impact research that
has recently been published. So although I think it is reasonable to
start by replicating many studies from, say, particular journals from
2008, I also think it is important to be flexible and focus also on
current research. For instance, suppose a study is published in
Psychological Science or in Science, and the stats look suspicious or
the finding is highly counter-intuitive. In that case I imagine a
network like this one could quickly run a series of replication
studies and convincingly refute or support the finding. This way the
network would function like a watchdog or fact-checkers, reminding
researchers that if they do not replicate their findings properly, the
field will. The prospect of quick retribution may make people more
careful before they seek to publish certain results.

Perhaps we can also do these replications in a way that gives credit
to the people who carried out the replications. For instance, the work
could be presented as a special issue or compendium of papers, where
the introduction, methods section, and general discussion/conclusion
sections are written by "OSF", but the results sections are
contributed as separate papers, each authored by whomever conducted
the work. So with five replications there are five results sections.
This will not take up much journal space.

Finally, I think it is essential that any replication study outlines
exactly what will be tested *before* any participants are tested. I
honestly believe this should be a ground rule for all replication
research. The biggest problem I have with current "replications" is
that they are published by labs who have a vested interest in seeing
them work out.

Cheers,
E.J.

--

[Brian Nosek]

I like these ideas.  The notion of a "rapid-response" replication team has appeal, though I would frame it somewhat more positively/optimistically than this.  

Rather than being a watchdog serving retribution for misbehaving researchers, I'd like to see this group evolve the normative understanding of replication research - that is is a good thing, not a threat to the original researcher.  My assumption is that almost all researchers are genuinely trying to do the best science they can.  The problem is in the incentive structures, not in the individual scientists.  A successful evolution of the model for scientific progress would be one in which researchers would be excited rather than threatened to have others try to replicate their research.  So, the rapid response concept could be applied to identify recent findings that seem important.  If OSF is explicitly trying to replicate research that appears important - and thus worth replicating - then it is a compliment to the researchers (our finding was worth replicating!), and a service for trying to affirm or identify potential problems in the groundbreaking findings.  

Making replication a positive for everyone would be a huge shift and contribution.  

[Roger Giner-Sorolla]

All these puzzle pieces fit together.

If you have an evaluation system weighted more towards article
citations than journal citations (see Hegarty's recent piece in PoPS)
then it's in your interest to aid and abet replications which will
cite your work. This would be even stronger if method section
citations were weighted more heavily in a citation count.

Replication is also not threatening if the original experiment is
"confirmed" by an aggregate result that points in the same direction,
and ultimately does so reliably. I think this may be one outcome of
the replication project, many studies replicating but not
significantly, and ultimately forcing researchers to either come down
off a strict .05 standard or accept that their results "don't
replicate"!

The field also needs to be smarter about truly paradigm-shifting
experiments. JPSP failed to see Bem's precog paper as something out of
the ordinary - if we all had faith in our methods we should be
treating it as our equivalent of the faster-than-light particles
rather than giving a cynical shrug. In fact I don't think there are
too many other findings that could challenge our assumptions (evidence
for the immortal soul maybe?) because we are so used to finding
boundary conditions, local reversals and exceptions to our effects.

[sheila miguez]

I am delurking to ask what people think of the possibility of a citizen science replication project. I know open science enthusiasts who would be interested.

I think one problem might be lack of credibility in results from lay enthusiasts due to sloppy methods, and I've been hoping to find open courseware to help assuage this problem. Not lucky yet, so I have been thinking this idea needs work. I have still have materials from when I was an undergrad, so I can try to pull together a self directed course of study.

I am wondering what the consensus would be with respect to this? Would this type of thing be of any use to the replication study? And would it help contribute to the overall health of the behavioral sciences?

[Brian Nosek]

Doing good science requires effective understanding and implementation of methods, procedures, analytic tools, reporting standards, etc.  But, it does not require credentials.  Credentials are a signal of training and experience (certainly useful things to have).  But, good science can be done by anyone that takes the time to understand what they are trying to do, and then do it well.  

It might be useful to have an "advisor" that is integrated in the field of study - if only to provide critique along the way and share the tacit/insider knowledge that can be useful for ensuring a project does not run afoul of the unspoken norms of the field.  

[Daniel Lakens]

I created a first 5 page draft version for a paper about this
replication value idea. I discussed the PPS special issue with EJ, but
all contributions there are already determined. This could be a normal
submission to PPS perhaps. I invited most people that commented on
this thread through Google Docs. Feel free to comment and suggest
changes, and feel free not to look at it if you are too busy ;)

[sheila miguez]

Response from Bargh on the paper (and the blog post!), with subsequent
engaging comments.

http://www.psychologytoday.com/blog/the-natural-unconscious/201203/nothing-in-their-heads/comments

On Fri, Jan 20, 2012 at 4:39 PM, Brian Nosek <no...@virginia.edu> wrote:
> http://blogs.discovermagazine.com/notrocketscience/2012/01/18/primed-by-expectations-%E2%80%93-why-a-classic-psychology-experiment-isn%E2%80%99t-what-it-seemed/
>
>

--
sheila

[Roger Giner-Sorolla]

Never thought I'd see the day when Psychology Today would have the
most provocative exchanges in scientific psychology ... (must be a
sign of the apocalypse, along with Cracked.com now having the most
thoughtful humor articles ...) it's all good though!

[Bobbie]

Er... If I may add my 4 cents...
1) Although the main framework of the special issue is set, other
articles on related topics may be placed in the same issue by... me.
2) I am still wishing people would post the replications -- success or
failure/published or non-published/real or conceptual on
psychfiledrawer.org
3) Some day I will tell the whole story about we almost did a big
thing at Perspective on the walking study....
4) And... I think that all citations in articles should have a
subscript as to why they are citing the previous article. This how
legal references work. It's important to know why a case cited
another. Did it support it? Overrule it? What? We should know why
one paper cites another -- like, is it a replication? I think this
could be merged into the replication value idea maybe.

[sheila miguez]

Even more interesting commentary on a followup post in the Discover blog.

http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-replication-bargh-psychology-study-doyen/

The reproducibility project gets name checked in a comment[1]. The
commenter writes, "[...] We will have not-fully-mature (to be polite)
researchers failing to replicate findings, and this may muddy the
waters as much as it clarifies them."

This is why I have not spoken up much in the mailing list despite
having an interest in participating in the project as a "citizen
scientist". I have a very idealistic hope that the hackerspace I am
part of will be able to run psychology experiments as a part of its
open science group, but I think there is some bootstrapping required
before we get to the point of having enough credibility to be useful.
I have a lot of thoughts on this, but don't want to degrade the signal
to noise ratio in the mailing list.

regards

[1] http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-replication-bargh-psychology-study-doyen/#comment-77262

--
sheila

[Roger Giner-Sorolla]

Sanjay Srivastava gets it just right IMO:

http://hardsci.wordpress.com/2012/03/12/some-reflections-on-the-bargh-doyen-elderly-walking-priming-brouhaha/

[Denny Borsboom]

I think the comparison to the Higgs particle is dubious. If the Bargh
effect is to be compared to anything in physics, it should be cold
fusion. Interestingly, the cold fusion effect was exposed as a fluke
within days after it was published. We're having this discussion 16
years after the original publication...
best
denny

--
Denny Borsboom
Department of Psychology

University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands

+31 20 525 6882
d.bor...@uva.nl
http://sites.google.com/site/borsboomdenny/dennyborsboom