Chronicle coverage
Roger Giner-Sorolla
Denny Borsboom
half?) of the results published in these three top psychology journals
can’t be replicated, it’s not going to reflect well on the field or on
the researchers whose papers didn’t pass the test."
I've been working on establishing a ball park number for the expected
number of successful replications. In my preliminary calculations the
probability of replicating a finding under reasonable assumptions,
given a uniform power of .8, hovers between around .6-.8. This makes
sense because even if all the results in the literature were correctly
refuted nulls, the probability of replicating them would equal power,
hence 80%. Most of the other forces at work (replicating nonrefuted
nulls, replicating fluke effects) tend to attenuate that number
downwards.
Thus finding 25% non-replications need not necessarily indicate
structural problems.
best
denny
--
Denny Borsboom
Department of Psychology
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands
+31 20 525 6882
d.bor...@uva.nl
http://sites.google.com/site/borsboomdenny/dennyborsboom
Brian Nosek
emphasizing the positives that we have been discussing persistently.
The negative angle must just have been too exciting to ignore.
This is discouraging and will take some work to undo the damage of the
messaging.
My apologies for being ineffective on this one.
Brian Nosek
if you care to do so):
-------------------
Tom -
I am disappointed with the negative tone of this post.
Reproducibility ("checking your work") is a central pillar of science.
Conducting a replication is not a threat to the integrity of the
publishing scientist or of the original work. It is an ordinary
practice to improve the confidence in scientific knowledge. The
scientist ideal is not to "be right" it is to "get it right."
Further a failure to replicate does NOT mean that the effect is false
or that the original researcher did something wrong. It *might* mean
that the effect is false, but there are many contributors to a lack of
reproducibility, and all of them are important to understand.
Most important, the Reproducibility Project is a project of
psychologists about psychology. We are investigating ourselves. The
sample of studies is just that - a sample. From that sample, we hope
to learn something about the population from which we drew it. That
population includes me and my collaborators. If the results suggest
low reproducibility, then I take it to mean as much about me and my
laboratory as it does about any article in the sample.
Science is self-critical. If there are problems, scientists aim to
identify them and do something about it. That is what I love about
science, and why I am a psychologist.
---------------
Brian Nosek
getting much too much credit for the project. It is totally fine with
me to be out front for any of the negative pushback (I can't get
fired, I checked.). But, I do worry that the crowdsourcing,
group-based effort is getting lost a bit.
If you have suggestions about how to shift that, please make them.
Also, if others comment in the Chronicle blog about other features -
e.g., that we work collaboratively with original authors to maximize
the quality of design, that most of the original authors have been
gracious and generous with their time (because they care too!) -- that
might help emphasize that this project is really a group project.
And, if you care to send any critical information or suggestions about
how I am handling this, please do. Public or private is fine.
Sheila Miguez
On Apr 18, 2012, at 9:20 AM, Brian Nosek wrote:
> gracious and generous with their time (because they care too!) -- that
> might help emphasize that this project is really a group project.
Encourage everyone to post progress updates and questions to the mailing list. If members have blogs, twitter accounts, or other such things, have a place that makes the activity easy to find. It would give observers a metric for engagement.
Joshua Foster
______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548
http://www.southalabama.edu/psychology/Faculty_Foster.htm
Eric-Jan Wagenmakers
people will perceive this or how they may feel about the scientific
status of psychology. If it's 40% it's 40%. This is interesting to
know. What other people do with this number is their business.
I feel the replication project is about replication, not about how we
best promote our discipline. My opinion is that there are many things
to improve in psychology. The fact that there are also many things to
improve in economics, biology, etc. is something that I'm not
immediately concerned with. If a patient is sick there needs to be a
diagnosis and a cure. The fact that other people may also be sick is
not very relevant.
So I don't think we need to get worried about negative press. These
are journalists, not scientists, and they'll say what they think is
needed to have other people read their stuff. Ultimately, whatever
they write is forgotten within a day.
Cheers,
E.J.
--
********************************************
WinBUGS workshop in Amsterdam: http://bayescourse.socsci.uva.nl
Eric-Jan Wagenmakers
Department of Psychological Methods, room 2.16
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands
Web: www.ejwagenmakers.com
Email: EJ.Wage...@gmail.com
Phone: (+31) 20 525 6420
“Man follows only phantoms.”
Pierre-Simon Laplace, last words
********************************************
Frank Farach
Frank J. Farach, Ph.D.
Postdoctoral Fellow
Center for Anxiety and Traumatic Stress
University of Washington
Department of Psychology
Guthrie Hall, Box 351525
Seattle, WA 98195-1525
Phone: (206) 616-2837
Fax: (206) 616-3156
Email: far...@u.washington.edu
Joshua Foster
either whether psychology replicates better or worse than other
disciplines. Replication information is useful regardless. But the
people who control the purse strings over here are looking for reasons
to reduce funding and providing them with low replication rates without
also providing proper context is just handing them a reason to nail
psychology. My nightmare scenario is that we simply report the rates
without providing proper context (i.e., comparisons to other
disciplines) and then some politician holds this study up as he or she
calls for the elimination of funding for this pseudoscience. Of course,
we would then argue that every field has issues with replication, but by
that point the story has taken off and we'd be fighting an uphill battle
for sure. I just don't think it's wise to operate in a scientific
bubble. Again, I would never argue that we should try to make psychology
look good or present only favorable comparisons. But we should make damn
sure that the people who read our research (and many will not be
scientists) have the proper context to evaluate our results.
______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548
http://www.southalabama.edu/psychology/Faculty_Foster.htm
>>> Eric-Jan Wagenmakers <ej.wage...@gmail.com> 4/18/2012 10:15 AM
Brian Nosek
Joshua Foster
http://www.nature.com/nature/journal/v483/n7391/full/483531a.html
______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548
http://www.southalabama.edu/psychology/Faculty_Foster.htm
>>> Brian Nosek <no...@virginia.edu> 4/18/2012 11:59 AM >>>
Simonsohn, Uri
As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are
too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations
at all.
I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours. She takes it with her to conference discussion, referee reports, decisions of whom to invite for talks, etc.
The argument, I guess, is that if one wants to be persuasive within the field, involving the press has the downside of being potentially counterproductive and the upside is not obvious.
My 2 cents, again, as mostly an outsider, I may be missing a lot of perspective on this,
Uri
Joshua Foster
______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548
http://www.southalabama.edu/psychology/Faculty_Foster.htm
>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>>
As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations at all.
I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours. She takes it with her to conference discussion, referee reports, decisions of whom to invite for talks, etc.
The argument, I guess, is that if one wants to be persuasive within the field, involving the press has the downside of being potentially counterproductive and the upside is not obvious.
My 2 cents, again, as mostly an outsider, I may be missing a lot of perspective on this,
Uri
From: openscienc...@googlegroups.com [mailto:openscienc...@googlegroups.com] On Behalf Of Frank Farach
Sent: Wednesday, April 18, 2012 12:20 PM
To: openscienc...@googlegroups.com
Subject: Re: [OpenScienceFramework] Chronicle coverage
The idea of benchmarking our replication metrics against other systematic replication projects is intriguing and might provide valuable context to the discussion, as Joshua mentioned. Is anyone aware of such efforts outside of psychology?
Negative press doesn't bother me much as long as it doesn't gain traction in the politics of funding for psychological science.
Frank
--
Frank J. Farach, Ph.D.
Postdoctoral Fellow
Center for Anxiety and Traumatic Stress
University of Washington
Department of Psychology
Guthrie Hall, Box 351525
Seattle, WA 98195-1525
Phone: (206) 616-2837
Fax: (206) 616-3156
Email: far...@u.washington.edu<mailto:far...@u.washington.edu>
On Wed, Apr 18, 2012 at 7:58 AM, Joshua Foster <fos...@usouthal.edu<mailto:fos...@usouthal.edu>> wrote:
I think it is reasonable to be concerned about how the results of our replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with replicability. Anyway, just something that this article brought to mind.
______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548<tel:%28251%29%20460-6548>
http://www.southalabama.edu/psychology/Faculty_Foster.htm
Jeffrey Spies
1) Why are the people who are critical about the project so concerned
that this is going to destroy psychology? It assumes that we won't
replicate at the rate we should. But why would that be the case? If
there wasn't a problem in the field, why would they be worried about
this in the first place? Intrinsic in their doubt is an assumption
that a problem exists! Perhaps they worry that we're bad scientists
or we're going to make stuff up, but why is that specific to this
group? I have a feeling that that's not what they're worried about.
2) We should continue communicating our work in an open, honest,
scientific way. Should we pay special attention to making the work
understandable even by a lay audience? Absolutely; but we always
should. This just happens to be a case where a lay audience is likely
watching more closely.
3) In the end, as most of you probably agree, there are ONLY positives
coming out of this project. We should praise psychology for being so
progressive that a group of scientists like those in the Open Science
Framework/Collaboration are willing to embark on such a project that
sacrifices personal gain for the good of science. We should praise
psychology that there is this esteem for such a vital piece of the
scientific process. And we should praise psychology for recognizing
the problems both broadly in science and in our field and not just
talking about it--or worse--not NOT talking about it. If nothing
replicates, that's good to know. If everything replicates, that's
good to know. It's only positive because it means we know more and, if
need be, can make improvements. We are not planning to have a
witch-hunt at the end of this--going after researchers whose studies
didn't replicate--we're going to seek collaboration with them (as we
already have been doing) in order to dig deeper into the substantive
problem space and figure out why there are discrepancies.
Jeff.
Jesse Chandler
affective tone that the findings leave with decision makers and
citizens will not correspond with the conclusions that we draw from
the findings. What is said needs to be thought through carefully, with
an eye to how people on the outside of the field, and inside it may
see it.
First, regardless of what we find, whether a problem exists or not is
a comparative question and the standard against which the success rate
is judged is malleable. While we may believe that any replication rate
that is close to 80% is ok, the finding can still be framed as "more
than 20% of findings are false." This may not be a deliberate
misrepresentation, many people have difficulty with the idea of "80%
accurate science" because science is perceived to be absolute. In
other words, it isn't obvious that there is social consensus on what
rate we "should" replicate at.
In a similar comparative vein, we all seem to assume that if we as a
field move first, people will compare our efforts to that of other
fields and conclude "they must be progressive for doing this" but for
all we know, people could also infer "there is vivid evidence for non-
replicability in psych, yet I can't recall any evidence of problems
with econ, sociology or biology. Psych must have a more serious
problem than they do." This is not a reason not to do this project,
but the concern that this perception could emerge is legitimate and
needs to be considered carefully.
I also agree with Uri that the press is not the proper venue for these
discussions, especially without data. Everything turning out ok is not
a good story, and reporters are incentivized to find another angle.
There are too many degrees of freedom within which the tale can be
told. This may lead to yet more damage, either directly, or indirectly
thorough researchers interpretation of this project as presented by
the press. Note in the recent piece that by merely associating this
project with Stapel and Bem, it could easily leave one with the
impression that these exemplars are representative of the studies that
fail to replicate. In other words, 80% of us are Stapels. One that
becomes the remembered message, it is not clear how to unsay it.
One thing that also requires some thought is the extent to which we
want to make the individual failures to replicate explicit. One option
might be to present only the aggregate data in the actual writeup,
with specific failures to replicate in the supplementary materials for
those who care (I am not interested in the opinions of people to lazy
to read the supplemental materials anyway). At the same time, I also
believe that refusal to share materials should be made explicit
because this is contrary to the core values of science.
Jesse
Roger Giner-Sorolla
Not everyone in the media will be as sympathetic as Siri Carpenter unfortunately.
I do think that when the final results come in we will have to handle the standard of "replication" with great care. One thing I don't think the project takes into account is the distinction between failure to replicate a manipulation and failure to replicate an effect. That is, if my sample doesn't even get angry from the anger manipulation, it doesn't indict the original findings so much as if they do get angry but the anger fails to increase accuracy in dart throwing. Lebel & Peters in Perspectives last year is required reading on this point. Of course, not all studies have manipulation checks, but maybe this is the point - we need to pay lots more attention to methods, or any replication attempt will dissolve into "well, you didn't put the coke can on the oscillator like we did."
Brian Nosek
the purpose, findings, and meaning effectively.
Just one point that differs from some of the comments: I believe that
science has a professional responsibility to engage and educate the
funding public. If the public doesn't understand the fundamentals of
the process and content, it is our problem not theirs. Most of the
concepts are not particularly hard, they are just unfamiliar.
I take my experience in doing (and observing) the media engagement
with implicit bias as a case in point. In 1990's public discussion
about bias and prejudice was way behind the science - still on "old
fashioned" prejudice ideas, discrimination is by "bad people", always
situated in intention, etc. With sustained engagement by many
scientists, the media coverage particularly has matured dramatically.
It is much closer now (on average) to the present scientific
understanding.
We don't need all scientists to engage and educate the public, but I
do think it is important that (the collective) we do it.
Russ Clay
In regard to the concern that finding a high type I error rate could be used for political purposes to defund psychological research; I would hope that somebody in the field who is in a position to get an audience with politicians and/or funding agencies would be quick to point out that the field that checks its own work should be viewed as a better investment for public tax dollars compared to a field which does not. Using the previous example of a 40% error rate, if that's what is found, then it is what it is. At least it is a known quantity. There is nothing to say that other scientific fields which have not pursued replication of their research would not find errors at the same rate or higher. Thus, I would expect the fact that the field is conducting Q&A on itself to make a better case for public funding compared to a field whose error rate is unknown. At least, I would argue that that is the point that should be made loud and clear.
Russ
Robert Smith
this message board.
Eric-Jan Wagenmakers
important discussion). Suppose we find that replication rates are
surprisingly low, despite our best efforts. This reflects poorly on
the field as a whole, however way you look at it -- if we cannot
replicate what is published, psychology fails the litmus test for
scientific credibility. I don't think that this will happen, but let's
assume.
So let's say we will establish that psychological findings can hardly
be replicated. Tax payers, represented by the government, may now feel
that it is wise to tighten the funding for fields in which few
findings actually replicate. I realize that it is never pleasant to
part with grant money, but wouldn't the tax payers have a point here?
As a tax player, how would you see your money allocated?
The defense that tightening funding for psychology is not fair because
other fields may *also* have a replicability problem is not
compelling, as it only suggest that funding should be cut for other
fields as well. We all like to have more money for research, but if we
find that psychology has a massive replicability problem it is
understandable that we receive less money from the government.
Apparently, we do not put the money to good use (again, I don't think
this is what our efforts will show, but this is certainly a line of
argument that a journalist could consider).
Suppose a police station runs an internal investigation into fraud,
and finds that 80% of officers take bribes. The argument that other
police stations may are also corrupt is not a good argument against
the state taking action for that particular police station.
So I see the pragmatics (i.e., we don't want to lose funding because
we are being self-critical, which is a good thing), but I am just not
sure whether the fear of seeing funding evaporate carries much weight
in the end.
I guess it is possible to argue the other way around as well: if
psychology is in a bad state we need more money to clean up our act.
Cheers,
E.J.
--
Brian Nosek
And, getting it right on the long-term will be accompanied by more funding overall because the knowledge is accurate and useful.
Sent from my iPhone
Joshua Foster
as you, E.J., then I wouldn't be worried about any of this. I agree with
everything you said, but especially the last sentence. People may
dismiss me as a pessimist, but I think psychologists are trying to do
the near impossible--that is, try to measure and manipulate the most
complicated thing in the known universe with crude instruments. If we're
right even some of the time, then in my mind that is a huge
accomplishment and worthy of continued effort.
______________________________
Joshua D. Foster, Ph.D.
Associate Professor
Psychology Department
University of South Alabama
Mobile, AL 36688
(251) 460-6548
www.joshuadfoster.com
>>> Eric-Jan Wagenmakers <ej.wage...@gmail.com> 04/19/12 3:09 PM >>>
Cheers,
E.J.
>> rate we "should" replic>> field move first, people will compare our efforts to that of other
>>> > Guthrie Hall, Box>>> > Email: far...@u.washington.edu<mailto:far...@u.washington.edu>
Denny Borsboom
(which they argue shows that even best theories eventually fall) leads
to the conclusion that all present theories are false as well. The
idea is known as pessimistic induction.
http://en.wikipedia.org/wiki/Pessimistic_induction
Although I don't personally buy the argument, I do think that a
realistic assessment of the state of affairs would suggest that most
of what we believe is probably wrong, in psychology and elsewhere.
Science is not a tea party. It is hard and unforgiving and it will
kill all your bedtime stories if your wait long enough. But it is the
most successful epistemic enterprise in the history of mankind, and I
am happy and proud to be a part of it.
There will certainly be people who try to misuse results any
investigation to their financial benefit, and the current study is no
exception. However, there are also many people (some of the with their
hands on a purse string) who understand the basic idea of science and
who will act to reward the movement towards openness that is currently
taking place.
Best
Denny
--
Denny Borsboom
Department of Psychology
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands
Rebecca Saxe
Even better, that inference is sometimes called "The Pessimistic Meta-Induction from the History of Science" - one of my favourite technical terms.
Rebecca