Really do not like the tone of "Wow this is going to destroy psychology and if the people involved in this project are nuanced about it they're just trying to cover their asses."
"But still. If it turns out that a sizable percentage (a quarter? half?) of the results published in these three top psychology journals can’t be replicated, it’s not going to reflect well on the field or on the researchers whose papers didn’t pass the test."
I've been working on establishing a ball park number for the expected number of successful replications. In my preliminary calculations the probability of replicating a finding under reasonable assumptions, given a uniform power of .8, hovers between around .6-.8. This makes sense because even if all the results in the literature were correctly refuted nulls, the probability of replicating them would equal power, hence 80%. Most of the other forces at work (replicating nonrefuted nulls, replicating fluke effects) tend to attenuate that number downwards.
Thus finding 25% non-replications need not necessarily indicate structural problems.
best denny
On Wed, Apr 18, 2012 at 1:59 PM, Roger Giner-Sorolla
> Really do not like the tone of "Wow this is going to destroy psychology and > if the people involved in this project are nuanced about it they're just > trying to cover their asses."
Ouch, I really don't like the tone either. I spent the interview emphasizing the positives that we have been discussing persistently. The negative angle must just have been too exciting to ignore.
This is discouraging and will take some work to undo the damage of the messaging.
My apologies for being ineffective on this one.
On Wed, Apr 18, 2012 at 4:59 AM, Roger Giner-Sorolla
> Really do not like the tone of "Wow this is going to destroy psychology and > if the people involved in this project are nuanced about it they're just > trying to cover their asses."
Here is a comment I posted on the Chronicle blog (do comment yourself if you care to do so):
------------------- Tom -
I am disappointed with the negative tone of this post. Reproducibility ("checking your work") is a central pillar of science. Conducting a replication is not a threat to the integrity of the publishing scientist or of the original work. It is an ordinary practice to improve the confidence in scientific knowledge. The scientist ideal is not to "be right" it is to "get it right."
Further a failure to replicate does NOT mean that the effect is false or that the original researcher did something wrong. It *might* mean that the effect is false, but there are many contributors to a lack of reproducibility, and all of them are important to understand.
Most important, the Reproducibility Project is a project of psychologists about psychology. We are investigating ourselves. The sample of studies is just that - a sample. From that sample, we hope to learn something about the population from which we drew it. That population includes me and my collaborators. If the results suggest low reproducibility, then I take it to mean as much about me and my laboratory as it does about any article in the sample.
Science is self-critical. If there are problems, scientists aim to identify them and do something about it. That is what I love about science, and why I am a psychologist. ---------------
On Wed, Apr 18, 2012 at 6:40 AM, Brian Nosek <no...@virginia.edu> wrote: > Ouch, I really don't like the tone either. I spent the interview > emphasizing the positives that we have been discussing persistently. > The negative angle must just have been too exciting to ignore.
> This is discouraging and will take some work to undo the damage of the > messaging.
>> Really do not like the tone of "Wow this is going to destroy psychology and >> if the people involved in this project are nuanced about it they're just >> trying to cover their asses."
The other thing that I am not quite sure how to address is that I am getting much too much credit for the project. It is totally fine with me to be out front for any of the negative pushback (I can't get fired, I checked.). But, I do worry that the crowdsourcing, group-based effort is getting lost a bit.
If you have suggestions about how to shift that, please make them. Also, if others comment in the Chronicle blog about other features - e.g., that we work collaboratively with original authors to maximize the quality of design, that most of the original authors have been gracious and generous with their time (because they care too!) -- that might help emphasize that this project is really a group project.
And, if you care to send any critical information or suggestions about how I am handling this, please do. Public or private is fine.
On Wed, Apr 18, 2012 at 7:02 AM, Brian Nosek <no...@virginia.edu> wrote: > Here is a comment I posted on the Chronicle blog (do comment yourself > if you care to do so):
> ------------------- > Tom -
> I am disappointed with the negative tone of this post. > Reproducibility ("checking your work") is a central pillar of science. > Conducting a replication is not a threat to the integrity of the > publishing scientist or of the original work. It is an ordinary > practice to improve the confidence in scientific knowledge. The > scientist ideal is not to "be right" it is to "get it right."
> Further a failure to replicate does NOT mean that the effect is false > or that the original researcher did something wrong. It *might* mean > that the effect is false, but there are many contributors to a lack of > reproducibility, and all of them are important to understand.
> Most important, the Reproducibility Project is a project of > psychologists about psychology. We are investigating ourselves. The > sample of studies is just that - a sample. From that sample, we hope > to learn something about the population from which we drew it. That > population includes me and my collaborators. If the results suggest > low reproducibility, then I take it to mean as much about me and my > laboratory as it does about any article in the sample.
> Science is self-critical. If there are problems, scientists aim to > identify them and do something about it. That is what I love about > science, and why I am a psychologist. > ---------------
> On Wed, Apr 18, 2012 at 6:40 AM, Brian Nosek <no...@virginia.edu> wrote: >> Ouch, I really don't like the tone either. I spent the interview >> emphasizing the positives that we have been discussing persistently. >> The negative angle must just have been too exciting to ignore.
>> This is discouraging and will take some work to undo the damage of the >> messaging.
>> My apologies for being ineffective on this one.
>>> Really do not like the tone of "Wow this is going to destroy psychology and >>> if the people involved in this project are nuanced about it they're just >>> trying to cover their asses."
> gracious and generous with their time (because they care too!) -- that > might help emphasize that this project is really a group project.
Encourage everyone to post progress updates and questions to the mailing list. If members have blogs, twitter accounts, or other such things, have a place that makes the activity easy to find. It would give observers a metric for engagement.
I think it is reasonable to be concerned about how the results of our replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with replicability. Anyway, just something that this article brought to mind.
I have a slightly different perspective. I don't care at all how other people will perceive this or how they may feel about the scientific status of psychology. If it's 40% it's 40%. This is interesting to know. What other people do with this number is their business.
I feel the replication project is about replication, not about how we best promote our discipline. My opinion is that there are many things to improve in psychology. The fact that there are also many things to improve in economics, biology, etc. is something that I'm not immediately concerned with. If a patient is sick there needs to be a diagnosis and a cure. The fact that other people may also be sick is not very relevant.
So I don't think we need to get worried about negative press. These are journalists, not scientists, and they'll say what they think is needed to have other people read their stuff. Ultimately, whatever they write is forgotten within a day.
Cheers, E.J.
On Wed, Apr 18, 2012 at 4:58 PM, Joshua Foster <fos...@usouthal.edu> wrote: > I think it is reasonable to be concerned about how the results of our replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with replicability. Anyway, just something that this article brought to mind.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548 > http://www.southalabama.edu/psychology/Faculty_Foster.htm
-- ******************************************** WinBUGS workshop in Amsterdam: http://bayescourse.socsci.uva.nl Eric-Jan Wagenmakers Department of Psychological Methods, room 2.16 University of Amsterdam Weesperplein 4 1018 XA Amsterdam The Netherlands
The idea of benchmarking our replication metrics against other systematic replication projects is intriguing and might provide valuable context to the discussion, as Joshua mentioned. Is anyone aware of such efforts outside of psychology?
Negative press doesn't bother me much as long as it doesn't gain traction in the politics of funding for psychological science.
Frank
-- Frank J. Farach, Ph.D. Postdoctoral Fellow Center for Anxiety and Traumatic Stress University of Washington Department of Psychology Guthrie Hall, Box 351525 Seattle, WA 98195-1525
On Wed, Apr 18, 2012 at 7:58 AM, Joshua Foster <fos...@usouthal.edu> wrote: > I think it is reasonable to be concerned about how the results of our > replication effort will be perceived by people outside of psychology. Let's > say we find that 40% of our results replicate. Personally, I wouldn't > consider that a failure of psychology and, indeed, it would be fairly > consistent with what other sciences seem to be finding. But without that > context it would be read by most "outsiders" as an indictment of > psychology. Because of this, I think it will be useful to present whatever > findings we get alongside what other sciences are finding (obviously, to > the extent that proper comparison groups exist). For example, we could cite > replication rates in the biomedical sciences as a comparison. The key will > be to do this consistently. Politicians, for example, are notorious for not > doing their research before making sweeping pronouncements. We cannot > assume that they or anyone else will bother to consider that other sciences > also have issues with replicability. Anyway, just something that this > article brought to mind.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548 > http://www.southalabama.edu/psychology/Faculty_Foster.htm
Let me be clear, from a scientific standpoint, it doesn't matter to me either whether psychology replicates better or worse than other disciplines. Replication information is useful regardless. But the people who control the purse strings over here are looking for reasons to reduce funding and providing them with low replication rates without also providing proper context is just handing them a reason to nail psychology. My nightmare scenario is that we simply report the rates without providing proper context (i.e., comparisons to other disciplines) and then some politician holds this study up as he or she calls for the elimination of funding for this pseudoscience. Of course, we would then argue that every field has issues with replication, but by that point the story has taken off and we'd be fighting an uphill battle for sure. I just don't think it's wise to operate in a scientific bubble. Again, I would never argue that we should try to make psychology look good or present only favorable comparisons. But we should make damn sure that the people who read our research (and many will not be scientists) have the proper context to evaluate our results.
>>> Eric-Jan Wagenmakers <ej.wagenmak...@gmail.com> 4/18/2012 10:15 AM
I have a slightly different perspective. I don't care at all how other people will perceive this or how they may feel about the scientific status of psychology. If it's 40% it's 40%. This is interesting to know. What other people do with this number is their business.
I feel the replication project is about replication, not about how we best promote our discipline. My opinion is that there are many things to improve in psychology. The fact that there are also many things to improve in economics, biology, etc. is something that I'm not immediately concerned with. If a patient is sick there needs to be a diagnosis and a cure. The fact that other people may also be sick is not very relevant.
So I don't think we need to get worried about negative press. These are journalists, not scientists, and they'll say what they think is needed to have other people read their stuff. Ultimately, whatever they write is forgotten within a day.
Cheers, E.J.
On Wed, Apr 18, 2012 at 4:58 PM, Joshua Foster <fos...@usouthal.edu> wrote:
> I think it is reasonable to be concerned about how the results of our
replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with replicability. Anyway, just something that this article brought to mind.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548 > http://www.southalabama.edu/psychology/Faculty_Foster.htm
-- ******************************************** WinBUGS workshop in Amsterdam: http://bayescourse.socsci.uva.nl Eric-Jan Wagenmakers Department of Psychological Methods, room 2.16 University of Amsterdam Weesperplein 4 1018 XA Amsterdam The Netherlands
On Frank and Joshua's point, there are two existing examples of reproducibility rates outside of psychology - one in cancer trials another in pharma. I'll retrieve and send (on my way to the airport) unless someone else can circulate first. They were mentioned briefly in an earlier post. And, upshot from them is alarming but clear in showing that - to the extent that there is an issue - it is not a local one. The replication rates in those studies (N's of 40-50 if I recall) were on the order of 10-20%.
On Wed, Apr 18, 2012 at 9:50 AM, Joshua Foster <fos...@usouthal.edu> wrote: > Let me be clear, from a scientific standpoint, it doesn't matter to me > either whether psychology replicates better or worse than other > disciplines. Replication information is useful regardless. But the > people who control the purse strings over here are looking for reasons > to reduce funding and providing them with low replication rates without > also providing proper context is just handing them a reason to nail > psychology. My nightmare scenario is that we simply report the rates > without providing proper context (i.e., comparisons to other > disciplines) and then some politician holds this study up as he or she > calls for the elimination of funding for this pseudoscience. Of course, > we would then argue that every field has issues with replication, but by > that point the story has taken off and we'd be fighting an uphill battle > for sure. I just don't think it's wise to operate in a scientific > bubble. Again, I would never argue that we should try to make psychology > look good or present only favorable comparisons. But we should make damn > sure that the people who read our research (and many will not be > scientists) have the proper context to evaluate our results.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548 > http://www.southalabama.edu/psychology/Faculty_Foster.htm
> >>> Eric-Jan Wagenmakers <ej.wagenmak...@gmail.com> 4/18/2012 10:15 AM
> I have a slightly different perspective. I don't care at all how other > people will perceive this or how they may feel about the scientific > status of psychology. If it's 40% it's 40%. This is interesting to > know. What other people do with this number is their business.
> I feel the replication project is about replication, not about how we > best promote our discipline. My opinion is that there are many things > to improve in psychology. The fact that there are also many things to > improve in economics, biology, etc. is something that I'm not > immediately concerned with. If a patient is sick there needs to be a > diagnosis and a cure. The fact that other people may also be sick is > not very relevant.
> So I don't think we need to get worried about negative press. These > are journalists, not scientists, and they'll say what they think is > needed to have other people read their stuff. Ultimately, whatever > they write is forgotten within a day.
> Cheers, > E.J.
> On Wed, Apr 18, 2012 at 4:58 PM, Joshua Foster <fos...@usouthal.edu> > wrote: > > I think it is reasonable to be concerned about how the results of our > replication effort will be perceived by people outside of psychology. > Let's say we find that 40% of our results replicate. Personally, I > wouldn't consider that a failure of psychology and, indeed, it would be > fairly consistent with what other sciences seem to be finding. But > without that context it would be read by most "outsiders" as an > indictment of psychology. Because of this, I think it will be useful to > present whatever findings we get alongside what other sciences are > finding (obviously, to the extent that proper comparison groups exist). > For example, we could cite replication rates in the biomedical sciences > as a comparison. The key will be to do this consistently. Politicians, > for example, are notorious for not doing their research before making > sweeping pronouncements. We cannot assume that they or anyone else will > bother to consider that other sciences also have issues with > replicability. Anyway, just something that this article brought to > mind.
> > ______________________________ > > Joshua D. Foster, Ph.D. > > Associate Professor > > Psychology Department > > University of South Alabama > > Mobile, AL 36688 > > (251) 460-6548 > > http://www.southalabama.edu/psychology/Faculty_Foster.htm
> -- > ******************************************** > WinBUGS workshop in Amsterdam: http://bayescourse.socsci.uva.nl > Eric-Jan Wagenmakers > Department of Psychological Methods, room 2.16 > University of Amsterdam > Weesperplein 4 > 1018 XA Amsterdam > The Netherlands
>>> Brian Nosek <no...@virginia.edu> 4/18/2012 11:59 AM >>>
On Frank and Joshua's point, there are two existing examples of reproducibility rates outside of psychology - one in cancer trials another in pharma. I'll retrieve and send (on my way to the airport) unless someone else can circulate first. They were mentioned briefly in an earlier post. And, upshot from them is alarming but clear in showing that - to the extent that there is an issue - it is not a local one. The replication rates in those studies (N's of 40-50 if I recall) were on the order of 10-20%.
On Wed, Apr 18, 2012 at 9:50 AM, Joshua Foster <fos...@usouthal.edu> wrote:
> Let me be clear, from a scientific standpoint, it doesn't matter to me > either whether psychology replicates better or worse than other > disciplines. Replication information is useful regardless. But the > people who control the purse strings over here are looking for reasons > to reduce funding and providing them with low replication rates without > also providing proper context is just handing them a reason to nail > psychology. My nightmare scenario is that we simply report the rates > without providing proper context (i.e., comparisons to other > disciplines) and then some politician holds this study up as he or she > calls for the elimination of funding for this pseudoscience. Of course, > we would then argue that every field has issues with replication, but by > that point the story has taken off and we'd be fighting an uphill battle > for sure. I just don't think it's wise to operate in a scientific > bubble. Again, I would never argue that we should try to make psychology > look good or present only favorable comparisons. But we should make damn > sure that the people who read our research (and many will not be > scientists) have the proper context to evaluate our results.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548 > http://www.southalabama.edu/psychology/Faculty_Foster.htm
> >>> Eric-Jan Wagenmakers <ej.wagenmak...@gmail.com> 4/18/2012 10:15 AM
> I have a slightly different perspective. I don't care at all how other > people will perceive this or how they may feel about the scientific > status of psychology. If it's 40% it's 40%. This is interesting to > know. What other people do with this number is their business.
> I feel the replication project is about replication, not about how we > best promote our discipline. My opinion is that there are many things > to improve in psychology. The fact that there are also many things to > improve in economics, biology, etc. is something that I'm not > immediately concerned with. If a patient is sick there needs to be a > diagnosis and a cure. The fact that other people may also be sick is > not very relevant.
> So I don't think we need to get worried about negative press. These > are journalists, not scientists, and they'll say what they think is > needed to have other people read their stuff. Ultimately, whatever > they write is forgotten within a day.
> Cheers, > E.J.
> On Wed, Apr 18, 2012 at 4:58 PM, Joshua Foster <fos...@usouthal.edu> > wrote: > > I think it is reasonable to be concerned about how the results of our > replication effort will be perceived by people outside of psychology. > Let's say we find that 40% of our results replicate. Personally, I > wouldn't consider that a failure of psychology and, indeed, it would be > fairly consistent with what other sciences seem to be finding. But > without that context it would be read by most "outsiders" as an > indictment of psychology. Because of this, I think it will be useful to > present whatever findings we get alongside what other sciences are > finding (obviously, to the extent that proper comparison groups exist). > For example, we could cite replication rates in the biomedical sciences > as a comparison. The key will be to do this consistently. Politicians, > for example, are notorious for not doing their research before making > sweeping pronouncements. We cannot assume that they or anyone else will > bother to consider that other sciences also have issues with > replicability. Anyway, just something that this article brought to > mind.
> > ______________________________ > > Joshua D. Foster, Ph.D. > > Associate Professor > > Psychology Department > > University of South Alabama > > Mobile, AL 36688 > > (251) 460-6548 > > http://www.southalabama.edu/psychology/Faculty_Foster.htm
> -- > ******************************************** > WinBUGS workshop in Amsterdam: http://bayescourse.socsci.uva.nl > Eric-Jan Wagenmakers > Department of Psychological Methods, room 2.16 > University of Amsterdam > Weesperplein 4 > 1018 XA Amsterdam > The Netherlands
As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations at all.
I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours. She takes it with her to conference discussion, referee reports, decisions of whom to invite for talks, etc.
The argument, I guess, is that if one wants to be persuasive within the field, involving the press has the downside of being potentially counterproductive and the upside is not obvious.
My 2 cents, again, as mostly an outsider, I may be missing a lot of perspective on this,
Uri
From: openscienceframework@googlegroups.com [mailto:openscienceframework@googlegroups.com] On Behalf Of Frank Farach Sent: Wednesday, April 18, 2012 12:20 PM To: openscienceframework@googlegroups.com Subject: Re: [OpenScienceFramework] Chronicle coverage
The idea of benchmarking our replication metrics against other systematic replication projects is intriguing and might provide valuable context to the discussion, as Joshua mentioned. Is anyone aware of such efforts outside of psychology?
Negative press doesn't bother me much as long as it doesn't gain traction in the politics of funding for psychological science.
Frank
-- Frank J. Farach, Ph.D. Postdoctoral Fellow Center for Anxiety and Traumatic Stress University of Washington Department of Psychology Guthrie Hall, Box 351525 Seattle, WA 98195-1525
On Wed, Apr 18, 2012 at 7:58 AM, Joshua Foster <fos...@usouthal.edu<mailto:fos...@usouthal.edu>> wrote: I think it is reasonable to be concerned about how the results of our replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with replicability. Anyway, just something that this article brought to mind.
______________________________ Joshua D. Foster, Ph.D. Associate Professor Psychology Department University of South Alabama Mobile, AL 36688 (251) 460-6548<tel:%28251%29%20460-6548> http://www.southalabama.edu/psychology/Faculty_Foster.htm
I also don't know how much negative press will affect funding decisions. Hopefully it won't, but I think we can do simple things like present comparison tables to try to prevent bad outcomes, not matter how unlikely. Maybe I'm particularly sensitive because in my state (Alabama), medical doctors (psychiatrists) are trying to remove the terms "diagnose" and' "treat" from the official job description of psychologists. They essentially argue that psychology lacks scientific credibility and that psychologists should not be entrusted to do these types of activities--that these activities should be left to the "real doctors." If we're going to present them a low replication rate for psychology, I want to at least make sure they see medicine's replication rate right next to it.
As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations at all.
I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours. She takes it with her to conference discussion, referee reports, decisions of whom to invite for talks, etc.
The argument, I guess, is that if one wants to be persuasive within the field, involving the press has the downside of being potentially counterproductive and the upside is not obvious.
My 2 cents, again, as mostly an outsider, I may be missing a lot of perspective on this,
Uri
From: openscienceframework@googlegroups.com [mailto:openscienceframework@googlegroups.com] On Behalf Of Frank Farach Sent: Wednesday, April 18, 2012 12:20 PM To: openscienceframework@googlegroups.com Subject: Re: [OpenScienceFramework] Chronicle coverage
The idea of benchmarking our replication metrics against other systematic replication projects is intriguing and might provide valuable context to the discussion, as Joshua mentioned. Is anyone aware of such efforts outside of psychology?
Negative press doesn't bother me much as long as it doesn't gain traction in the politics of funding for psychological science.
Frank
-- Frank J. Farach, Ph.D. Postdoctoral Fellow Center for Anxiety and Traumatic Stress University of Washington Department of Psychology Guthrie Hall, Box 351525 Seattle, WA 98195-1525
On Wed, Apr 18, 2012 at 7:58 AM, Joshua Foster <fos...@usouthal.edu<mailto:fos...@usouthal.edu>> wrote: I think it is reasonable to be concerned about how the results of our replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with replicability. Anyway, just something that this article brought to mind.
______________________________ Joshua D. Foster, Ph.D. Associate Professor Psychology Department University of South Alabama Mobile, AL 36688 (251) 460-6548<tel:%28251%29%20460-6548> http://www.southalabama.edu/psychology/Faculty_Foster.htm
I have just a few comments to add to this discussion:
1) Why are the people who are critical about the project so concerned that this is going to destroy psychology? It assumes that we won't replicate at the rate we should. But why would that be the case? If there wasn't a problem in the field, why would they be worried about this in the first place? Intrinsic in their doubt is an assumption that a problem exists! Perhaps they worry that we're bad scientists or we're going to make stuff up, but why is that specific to this group? I have a feeling that that's not what they're worried about.
2) We should continue communicating our work in an open, honest, scientific way. Should we pay special attention to making the work understandable even by a lay audience? Absolutely; but we always should. This just happens to be a case where a lay audience is likely watching more closely.
3) In the end, as most of you probably agree, there are ONLY positives coming out of this project. We should praise psychology for being so progressive that a group of scientists like those in the Open Science Framework/Collaboration are willing to embark on such a project that sacrifices personal gain for the good of science. We should praise psychology that there is this esteem for such a vital piece of the scientific process. And we should praise psychology for recognizing the problems both broadly in science and in our field and not just talking about it--or worse--not NOT talking about it. If nothing replicates, that's good to know. If everything replicates, that's good to know. It's only positive because it means we know more and, if need be, can make improvements. We are not planning to have a witch-hunt at the end of this--going after researchers whose studies didn't replicate--we're going to seek collaboration with them (as we already have been doing) in order to dig deeper into the substantive problem space and figure out why there are discrepancies.
On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> wrote: > I also don't know how much negative press will affect funding decisions. Hopefully it won't, but I think we can do simple things like present comparison tables to try to prevent bad outcomes, not matter how unlikely. Maybe I'm particularly sensitive because in my state (Alabama), medical doctors (psychiatrists) are trying to remove the terms "diagnose" and' "treat" from the official job description of psychologists. They essentially argue that psychology lacks scientific credibility and that psychologists should not be entrusted to do these types of activities--that these activities should be left to the "real doctors." If we're going to present them a low replication rate for psychology, I want to at least make sure they see medicine's replication rate right next to it.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548 > http://www.southalabama.edu/psychology/Faculty_Foster.htm
>>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>> > As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations at all.
> I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
> I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
> So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours. She takes it with her to conference discussion, referee reports, decisions of whom to invite for talks, etc.
> The argument, I guess, is that if one wants to be persuasive within the field, involving the press has the downside of being potentially counterproductive and the upside is not obvious.
> My 2 cents, again, as mostly an outsider, I may be missing a lot of perspective on this,
> Uri
> From: openscienceframework@googlegroups.com [mailto:openscienceframework@googlegroups.com] On Behalf Of Frank Farach > Sent: Wednesday, April 18, 2012 12:20 PM > To: openscienceframework@googlegroups.com > Subject: Re: [OpenScienceFramework] Chronicle coverage
> The idea of benchmarking our replication metrics against other systematic replication projects is intriguing and might provide valuable context to the discussion, as Joshua mentioned. Is anyone aware of such efforts outside of psychology?
> Negative press doesn't bother me much as long as it doesn't gain traction in the politics of funding for psychological science.
> Frank
> -- > Frank J. Farach, Ph.D. > Postdoctoral Fellow > Center for Anxiety and Traumatic Stress > University of Washington > Department of Psychology > Guthrie Hall, Box 351525 > Seattle, WA 98195-1525
> On Wed, Apr 18, 2012 at 7:58 AM, Joshua Foster <fos...@usouthal.edu<mailto:fos...@usouthal.edu>> wrote: > I think it is reasonable to be concerned about how the results of our replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with replicability. Anyway, just something that this article brought to mind.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548<tel:%28251%29%20460-6548> > http://www.southalabama.edu/psychology/Faculty_Foster.htm
I think that there is a legitimate concern that the perception and
affective tone that the findings leave with decision makers and
citizens will not correspond with the conclusions that we draw from
the findings. What is said needs to be thought through carefully, with
an eye to how people on the outside of the field, and inside it may
see it.
First, regardless of what we find, whether a problem exists or not is
a comparative question and the standard against which the success rate
is judged is malleable. While we may believe that any replication rate
that is close to 80% is ok, the finding can still be framed as "more
than 20% of findings are false." This may not be a deliberate
misrepresentation, many people have difficulty with the idea of "80%
accurate science" because science is perceived to be absolute. In
other words, it isn't obvious that there is social consensus on what
rate we "should" replicate at.
In a similar comparative vein, we all seem to assume that if we as a
field move first, people will compare our efforts to that of other
fields and conclude "they must be progressive for doing this" but for
all we know, people could also infer "there is vivid evidence for non-
replicability in psych, yet I can't recall any evidence of problems
with econ, sociology or biology. Psych must have a more serious
problem than they do." This is not a reason not to do this project,
but the concern that this perception could emerge is legitimate and
needs to be considered carefully.
I also agree with Uri that the press is not the proper venue for these
discussions, especially without data. Everything turning out ok is not
a good story, and reporters are incentivized to find another angle.
There are too many degrees of freedom within which the tale can be
told. This may lead to yet more damage, either directly, or indirectly
thorough researchers interpretation of this project as presented by
the press. Note in the recent piece that by merely associating this
project with Stapel and Bem, it could easily leave one with the
impression that these exemplars are representative of the studies that
fail to replicate. In other words, 80% of us are Stapels. One that
becomes the remembered message, it is not clear how to unsay it.
One thing that also requires some thought is the extent to which we
want to make the individual failures to replicate explicit. One option
might be to present only the aggregate data in the actual writeup,
with specific failures to replicate in the supplementary materials for
those who care (I am not interested in the opinions of people to lazy
to read the supplemental materials anyway). At the same time, I also
believe that refusal to share materials should be made explicit
because this is contrary to the core values of science.
Jesse
On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote:
> I have just a few comments to add to this discussion:
> 1) Why are the people who are critical about the project so concerned
> that this is going to destroy psychology? It assumes that we won't
> replicate at the rate we should. But why would that be the case? If
> there wasn't a problem in the field, why would they be worried about
> this in the first place? Intrinsic in their doubt is an assumption
> that a problem exists! Perhaps they worry that we're bad scientists
> or we're going to make stuff up, but why is that specific to this
> group? I have a feeling that that's not what they're worried about.
> 2) We should continue communicating our work in an open, honest,
> scientific way. Should we pay special attention to making the work
> understandable even by a lay audience? Absolutely; but we always
> should. This just happens to be a case where a lay audience is likely
> watching more closely.
> 3) In the end, as most of you probably agree, there are ONLY positives
> coming out of this project. We should praise psychology for being so
> progressive that a group of scientists like those in the Open Science
> Framework/Collaboration are willing to embark on such a project that
> sacrifices personal gain for the good of science. We should praise
> psychology that there is this esteem for such a vital piece of the
> scientific process. And we should praise psychology for recognizing
> the problems both broadly in science and in our field and not just
> talking about it--or worse--not NOT talking about it. If nothing
> replicates, that's good to know. If everything replicates, that's
> good to know. It's only positive because it means we know more and, if
> need be, can make improvements. We are not planning to have a
> witch-hunt at the end of this--going after researchers whose studies
> didn't replicate--we're going to seek collaboration with them (as we
> already have been doing) in order to dig deeper into the substantive
> problem space and figure out why there are discrepancies.
> Jeff.
> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> wrote:
> > I also don't know how much negative press will affect funding decisions. Hopefully it won't, but I think we can do simple things like present comparison tables to try to prevent bad outcomes, not matter how unlikely. Maybe I'm particularly sensitive because in my state (Alabama), medical doctors (psychiatrists) are trying to remove the terms "diagnose" and' "treat" from the official job description of psychologists. They essentially argue that psychology lacks scientific credibility and that psychologists should not be entrusted to do these types of activities--that these activities should be left to the "real doctors." If we're going to present them a low replication rate for psychology, I want to at least make sure they see medicine's replication rate right next to it.
> > ______________________________
> > Joshua D. Foster, Ph.D.
> > Associate Professor
> > Psychology Department
> > University of South Alabama
> > Mobile, AL 36688
> > (251) 460-6548
> >http://www.southalabama.edu/psychology/Faculty_Foster.htm
> >>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>>
> > As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations at all.
> > I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
> > I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
> > So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours. She takes it with her to conference discussion, referee reports, decisions of whom to invite for talks, etc.
> > The argument, I guess, is that if one wants to be persuasive within the field, involving the press has the downside of being potentially counterproductive and the upside is not obvious.
> > My 2 cents, again, as mostly an outsider, I may be missing a lot of perspective on this,
> > Uri
> > From: openscienceframework@googlegroups.com [mailto:openscienceframework@googlegroups.com] On Behalf Of Frank Farach
> > Sent: Wednesday, April 18, 2012 12:20 PM
> > To: openscienceframework@googlegroups.com
> > Subject: Re: [OpenScienceFramework] Chronicle coverage
> > The idea of benchmarking our replication metrics against other systematic replication projects is intriguing and might provide valuable context to the discussion, as Joshua mentioned. Is anyone aware of such efforts outside of psychology?
> > Negative press doesn't bother me much as long as it doesn't gain traction in the politics of funding for psychological science.
> > Frank
> > --
> > Frank J. Farach, Ph.D.
> > Postdoctoral Fellow
> > Center for Anxiety and Traumatic Stress
> > University of Washington
> > Department of Psychology
> > Guthrie Hall, Box 351525
> > Seattle, WA 98195-1525
> > On Wed, Apr 18, 2012 at 7:58 AM, Joshua Foster <fos...@usouthal.edu<mailto:fos...@usouthal.edu>> wrote:
> > I think it is reasonable to be concerned about how the results of our replication effort will be perceived by people outside of psychology. Let's say we find that 40% of our results replicate. Personally, I wouldn't consider that a failure of psychology and, indeed, it would be fairly consistent with what other sciences seem to be finding. But without that context it would be read by most "outsiders" as an indictment of psychology. Because of this, I think it will be useful to present whatever findings we get alongside what other sciences are finding (obviously, to the extent that proper comparison groups exist). For example, we could cite replication rates in the biomedical sciences as a comparison. The key will be to do this consistently. Politicians, for example, are notorious for not doing their research before making sweeping pronouncements. We cannot assume that they or anyone else will bother to consider that other sciences also have issues with
To me it's clear from the article that Brian was doing his best to present the project realistically and the journalist was trying just as hard to undercut Brian's efforts, in order to present it as something hostile and doom-laden.
Not everyone in the media will be as sympathetic as Siri Carpenter unfortunately.
I do think that when the final results come in we will have to handle the standard of "replication" with great care. One thing I don't think the project takes into account is the distinction between failure to replicate a manipulation and failure to replicate an effect. That is, if my sample doesn't even get angry from the anger manipulation, it doesn't indict the original findings so much as if they do get angry but the anger fails to increase accuracy in dart throwing. Lebel & Peters in Perspectives last year is required reading on this point. Of course, not all studies have manipulation checks, but maybe this is the point - we need to pay lots more attention to methods, or any replication attempt will dissolve into "well, you didn't put the coke can on the oscillator like we did."
Many good points being raised in this discussion about how to present the purpose, findings, and meaning effectively.
Just one point that differs from some of the comments: I believe that science has a professional responsibility to engage and educate the funding public. If the public doesn't understand the fundamentals of the process and content, it is our problem not theirs. Most of the concepts are not particularly hard, they are just unfamiliar.
I take my experience in doing (and observing) the media engagement with implicit bias as a case in point. In 1990's public discussion about bias and prejudice was way behind the science - still on "old fashioned" prejudice ideas, discrimination is by "bad people", always situated in intention, etc. With sustained engagement by many scientists, the media coverage particularly has matured dramatically. It is much closer now (on average) to the present scientific understanding.
We don't need all scientists to engage and educate the public, but I do think it is important that (the collective) we do it.
<j.j.b.chand...@gmail.com> wrote: > I think that there is a legitimate concern that the perception and > affective tone that the findings leave with decision makers and > citizens will not correspond with the conclusions that we draw from > the findings. What is said needs to be thought through carefully, with > an eye to how people on the outside of the field, and inside it may > see it.
> First, regardless of what we find, whether a problem exists or not is > a comparative question and the standard against which the success rate > is judged is malleable. While we may believe that any replication rate > that is close to 80% is ok, the finding can still be framed as "more > than 20% of findings are false." This may not be a deliberate > misrepresentation, many people have difficulty with the idea of "80% > accurate science" because science is perceived to be absolute. In > other words, it isn't obvious that there is social consensus on what > rate we "should" replicate at.
> In a similar comparative vein, we all seem to assume that if we as a > field move first, people will compare our efforts to that of other > fields and conclude "they must be progressive for doing this" but for > all we know, people could also infer "there is vivid evidence for non- > replicability in psych, yet I can't recall any evidence of problems > with econ, sociology or biology. Psych must have a more serious > problem than they do." This is not a reason not to do this project, > but the concern that this perception could emerge is legitimate and > needs to be considered carefully.
> I also agree with Uri that the press is not the proper venue for these > discussions, especially without data. Everything turning out ok is not > a good story, and reporters are incentivized to find another angle. > There are too many degrees of freedom within which the tale can be > told. This may lead to yet more damage, either directly, or indirectly > thorough researchers interpretation of this project as presented by > the press. Note in the recent piece that by merely associating this > project with Stapel and Bem, it could easily leave one with the > impression that these exemplars are representative of the studies that > fail to replicate. In other words, 80% of us are Stapels. One that > becomes the remembered message, it is not clear how to unsay it.
> One thing that also requires some thought is the extent to which we > want to make the individual failures to replicate explicit. One option > might be to present only the aggregate data in the actual writeup, > with specific failures to replicate in the supplementary materials for > those who care (I am not interested in the opinions of people to lazy > to read the supplemental materials anyway). At the same time, I also > believe that refusal to share materials should be made explicit > because this is contrary to the core values of science.
> Jesse
> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote: >> I have just a few comments to add to this discussion:
>> 1) Why are the people who are critical about the project so concerned >> that this is going to destroy psychology? It assumes that we won't >> replicate at the rate we should. But why would that be the case? If >> there wasn't a problem in the field, why would they be worried about >> this in the first place? Intrinsic in their doubt is an assumption >> that a problem exists! Perhaps they worry that we're bad scientists >> or we're going to make stuff up, but why is that specific to this >> group? I have a feeling that that's not what they're worried about.
>> 2) We should continue communicating our work in an open, honest, >> scientific way. Should we pay special attention to making the work >> understandable even by a lay audience? Absolutely; but we always >> should. This just happens to be a case where a lay audience is likely >> watching more closely.
>> 3) In the end, as most of you probably agree, there are ONLY positives >> coming out of this project. We should praise psychology for being so >> progressive that a group of scientists like those in the Open Science >> Framework/Collaboration are willing to embark on such a project that >> sacrifices personal gain for the good of science. We should praise >> psychology that there is this esteem for such a vital piece of the >> scientific process. And we should praise psychology for recognizing >> the problems both broadly in science and in our field and not just >> talking about it--or worse--not NOT talking about it. If nothing >> replicates, that's good to know. If everything replicates, that's >> good to know. It's only positive because it means we know more and, if >> need be, can make improvements. We are not planning to have a >> witch-hunt at the end of this--going after researchers whose studies >> didn't replicate--we're going to seek collaboration with them (as we >> already have been doing) in order to dig deeper into the substantive >> problem space and figure out why there are discrepancies.
>> Jeff.
>> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> wrote: >> > I also don't know how much negative press will affect funding decisions. Hopefully it won't, but I think we can do simple things like present comparison tables to try to prevent bad outcomes, not matter how unlikely. Maybe I'm particularly sensitive because in my state (Alabama), medical doctors (psychiatrists) are trying to remove the terms "diagnose" and' "treat" from the official job description of psychologists. They essentially argue that psychology lacks scientific credibility and that psychologists should not be entrusted to do these types of activities--that these activities should be left to the "real doctors." If we're going to present them a low replication rate for psychology, I want to at least make sure they see medicine's replication rate right next to it.
>> > ______________________________ >> > Joshua D. Foster, Ph.D. >> > Associate Professor >> > Psychology Department >> > University of South Alabama >> > Mobile, AL 36688 >> > (251) 460-6548 >> >http://www.southalabama.edu/psychology/Faculty_Foster.htm
>> >>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>> >> > As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations at all.
>> > I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
>> > I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
>> > So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours. She takes it with her to conference discussion, referee reports, decisions of whom to invite for talks, etc.
>> > The argument, I guess, is that if one wants to be persuasive within the field, involving the press has the downside of being potentially counterproductive and the upside is not obvious.
>> > My 2 cents, again, as mostly an outsider, I may be missing a lot of perspective on this,
>> > Uri
>> > From: openscienceframework@googlegroups.com [mailto:openscienceframework@googlegroups.com] On Behalf Of Frank Farach >> > Sent: Wednesday, April 18, 2012 12:20 PM >> > To: openscienceframework@googlegroups.com >> > Subject: Re: [OpenScienceFramework] Chronicle coverage
>> > The idea of benchmarking our replication metrics against other systematic replication projects is intriguing and might provide valuable context to the discussion, as Joshua mentioned. Is anyone aware of such efforts outside of psychology?
>> > Negative press doesn't bother me much as long as it doesn't gain traction in the politics of funding for psychological science.
>> > Frank
>> > -- >> > Frank J. Farach, Ph.D. >> > Postdoctoral Fellow >> > Center for Anxiety and Traumatic Stress >> > University of Washington >> > Department of Psychology >> > Guthrie Hall, Box 351525 >> > Seattle, WA 98195-1525
In regard to the concern that finding a high type I error rate could be used for political purposes to defund psychological research; I would hope that somebody in the field who is in a position to get an audience with politicians and/or funding agencies would be quick to point out that the field that checks its own work should be viewed as a better investment for public tax dollars compared to a field which does not. Using the previous example of a 40% error rate, if that's what is found, then it is what it is. At least it is a known quantity. There is nothing to say that other scientific fields which have not pursued replication of their research would not find errors at the same rate or higher. Thus, I would expect the fact that the field is conducting Q&A on itself to make a better case for public funding compared to a field whose error rate is unknown. At least, I would argue that that is the point that should be made loud and clear.
-----Original Message----- From: openscienceframework@googlegroups.com [mailto:openscienceframework@googlegroups.com] On Behalf Of Brian Nosek Sent: Thursday, April 19, 2012 7:32 AM To: openscienceframework@googlegroups.com Subject: Re: [OpenScienceFramework] Re: Chronicle coverage
Many good points being raised in this discussion about how to present the purpose, findings, and meaning effectively.
Just one point that differs from some of the comments: I believe that science has a professional responsibility to engage and educate the funding public. If the public doesn't understand the fundamentals of the process and content, it is our problem not theirs. Most of the concepts are not particularly hard, they are just unfamiliar.
I take my experience in doing (and observing) the media engagement with implicit bias as a case in point. In 1990's public discussion about bias and prejudice was way behind the science - still on "old fashioned" prejudice ideas, discrimination is by "bad people", always situated in intention, etc. With sustained engagement by many scientists, the media coverage particularly has matured dramatically. It is much closer now (on average) to the present scientific understanding.
We don't need all scientists to engage and educate the public, but I do think it is important that (the collective) we do it.
On Wed, Apr 18, 2012 at 7:33 PM, Jesse Chandler <j.j.b.chand...@gmail.com> wrote: > I think that there is a legitimate concern that the perception and > affective tone that the findings leave with decision makers and > citizens will not correspond with the conclusions that we draw from > the findings. What is said needs to be thought through carefully, with > an eye to how people on the outside of the field, and inside it may > see it.
> First, regardless of what we find, whether a problem exists or not is > a comparative question and the standard against which the success rate > is judged is malleable. While we may believe that any replication rate > that is close to 80% is ok, the finding can still be framed as "more > than 20% of findings are false." This may not be a deliberate > misrepresentation, many people have difficulty with the idea of "80% > accurate science" because science is perceived to be absolute. In > other words, it isn't obvious that there is social consensus on what > rate we "should" replicate at.
> In a similar comparative vein, we all seem to assume that if we as a > field move first, people will compare our efforts to that of other > fields and conclude "they must be progressive for doing this" but for > all we know, people could also infer "there is vivid evidence for non- > replicability in psych, yet I can't recall any evidence of problems > with econ, sociology or biology. Psych must have a more serious > problem than they do." This is not a reason not to do this project, > but the concern that this perception could emerge is legitimate and > needs to be considered carefully.
> I also agree with Uri that the press is not the proper venue for these > discussions, especially without data. Everything turning out ok is not > a good story, and reporters are incentivized to find another angle. > There are too many degrees of freedom within which the tale can be > told. This may lead to yet more damage, either directly, or indirectly > thorough researchers interpretation of this project as presented by > the press. Note in the recent piece that by merely associating this > project with Stapel and Bem, it could easily leave one with the > impression that these exemplars are representative of the studies that > fail to replicate. In other words, 80% of us are Stapels. One that > becomes the remembered message, it is not clear how to unsay it.
> One thing that also requires some thought is the extent to which we > want to make the individual failures to replicate explicit. One option > might be to present only the aggregate data in the actual writeup, > with specific failures to replicate in the supplementary materials for > those who care (I am not interested in the opinions of people to lazy > to read the supplemental materials anyway). At the same time, I also > believe that refusal to share materials should be made explicit > because this is contrary to the core values of science.
> Jesse
> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote: >> I have just a few comments to add to this discussion:
>> 1) Why are the people who are critical about the project so concerned >> that this is going to destroy psychology? It assumes that we won't >> replicate at the rate we should. But why would that be the case? If >> there wasn't a problem in the field, why would they be worried about >> this in the first place? Intrinsic in their doubt is an assumption >> that a problem exists! Perhaps they worry that we're bad scientists >> or we're going to make stuff up, but why is that specific to this >> group? I have a feeling that that's not what they're worried about.
>> 2) We should continue communicating our work in an open, honest, >> scientific way. Should we pay special attention to making the work >> understandable even by a lay audience? Absolutely; but we always >> should. This just happens to be a case where a lay audience is >> likely watching more closely.
>> 3) In the end, as most of you probably agree, there are ONLY >> positives coming out of this project. We should praise psychology >> for being so progressive that a group of scientists like those in the >> Open Science Framework/Collaboration are willing to embark on such a >> project that sacrifices personal gain for the good of science. We >> should praise psychology that there is this esteem for such a vital >> piece of the scientific process. And we should praise psychology for >> recognizing the problems both broadly in science and in our field and >> not just talking about it--or worse--not NOT talking about it. If >> nothing replicates, that's good to know. If everything replicates, >> that's good to know. It's only positive because it means we know more >> and, if need be, can make improvements. We are not planning to have >> a witch-hunt at the end of this--going after researchers whose >> studies didn't replicate--we're going to seek collaboration with them >> (as we already have been doing) in order to dig deeper into the >> substantive problem space and figure out why there are discrepancies.
>> Jeff.
>> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> wrote: >> > I also don't know how much negative press will affect funding decisions. Hopefully it won't, but I think we can do simple things like present comparison tables to try to prevent bad outcomes, not matter how unlikely. Maybe I'm particularly sensitive because in my state (Alabama), medical doctors (psychiatrists) are trying to remove the terms "diagnose" and' "treat" from the official job description of psychologists. They essentially argue that psychology lacks scientific credibility and that psychologists should not be entrusted to do these types of activities--that these activities should be left to the "real doctors." If we're going to present them a low replication rate for psychology, I want to at least make sure they see medicine's replication rate right next to it.
>> > ______________________________ >> > Joshua D. Foster, Ph.D. >> > Associate Professor >> > Psychology Department >> > University of South Alabama >> > Mobile, AL 36688 >> > (251) 460-6548 >> >http://www.southalabama.edu/psychology/Faculty_Foster.htm
>> >>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>> >> > As somewhat of an outsider to the replication project, my view is that it is best not to get too involved with the press on these issues, the incentives are too misaligned. We want the best science in the long term, they want sensationalist stories that will sell papers tomorrow. We care about the reputations of the individuals being replicated and of those doing the replicating, they do not care about reputations at all.
>> > I agree with EJ that nobody will remember the stories within 48 hours, and I am a bit skeptical that such stories really have much of an impact on funding (though little evidence behind that so maybe just being optimistic).
>> > I worry quite a bit, however, about how the popular press distorts the dialogue within psychology.
>> > So take a psychologist who has not thought too much about replications, methodology, false-positive, fabrication, etc. (i.e., the modal scientist) And that psychologist reads about replication attempts (or fabrication or whatever) in the Times, if the tone is not perfect, if she feels her field is under attack, she becomes negatively predisposed to the endeveour of open science, replication, skepticism, etc, and she does NOT forget about it within 72 hours.
> I think that there is a legitimate concern that the perception and > affective tone that the findings leave with decision makers and > citizens will not correspond with the conclusions that we draw from > the findings. What is said needs to be thought through carefully, with > an eye to how people on the outside of the field, and inside it may > see it.
> First, regardless of what we find, whether a problem exists or not is > a comparative question and the standard against which the success rate > is judged is malleable. While we may believe that any replication rate > that is close to 80% is ok, the finding can still be framed as "more > than 20% of findings are false." This may not be a deliberate > misrepresentation, many people have difficulty with the idea of "80% > accurate science" because science is perceived to be absolute. In > other words, it isn't obvious that there is social consensus on what > rate we "should" replicate at.
> In a similar comparative vein, we all seem to assume that if we as a > field move first, people will compare our efforts to that of other > fields and conclude "they must be progressive for doing this" but for > all we know, people could also infer "there is vivid evidence for non- > replicability in psych, yet I can't recall any evidence of problems > with econ, sociology or biology. Psych must have a more serious > problem than they do." This is not a reason not to do this project, > but the concern that this perception could emerge is legitimate and > needs to be considered carefully.
> I also agree with Uri that the press is not the proper venue for these > discussions, especially without data. Everything turning out ok is not > a good story, and reporters are incentivized to find another angle. > There are too many degrees of freedom within which the tale can be > told. This may lead to yet more damage, either directly, or indirectly > thorough researchers interpretation of this project as presented by > the press. Note in the recent piece that by merely associating this > project with Stapel and Bem, it could easily leave one with the > impression that these exemplars are representative of the studies that > fail to replicate. In other words, 80% of us are Stapels. One that > becomes the remembered message, it is not clear how to unsay it.
> One thing that also requires some thought is the extent to which we > want to make the individual failures to replicate explicit. One option > might be to present only the aggregate data in the actual writeup, > with specific failures to replicate in the supplementary materials for > those who care (I am not interested in the opinions of people to lazy > to read the supplemental materials anyway). At the same time, I also > believe that refusal to share materials should be made explicit > because this is contrary to the core values of science.
> Jesse
> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote: >> I have just a few comments to add to this discussion:
>> 1) Why are the people who are critical about the project so concerned >> that this is going to destroy psychology? It assumes that we won't >> replicate at the rate we should. But why would that be the case? If >> there wasn't a problem in the field, why would they be worried about >> this in the first place? Intrinsic in their doubt is an assumption >> that a problem exists! Perhaps they worry that we're bad scientists >> or we're going to make stuff up, but why is that specific to this >> group? I have a feeling that that's not what they're worried about.
>> 2) We should continue communicating our work in an open, honest, >> scientific way. Should we pay special attention to making the work >> understandable even by a lay audience? Absolutely; but we always >> should. This just happens to be a case where a lay audience is likely >> watching more closely.
>> 3) In the end, as most of you probably agree, there are ONLY positives >> coming out of this project. We should praise psychology for being so >> progressive that a group of scientists like those in the Open Science >> Framework/Collaboration are willing to embark on such a project that >> sacrifices personal gain for the good of science. We should praise >> psychology that there is this esteem for such a vital piece of the >> scientific process. And we should praise psychology for recognizing >> the problems both broadly in science and in our field and not just >> talking about it--or worse--not NOT talking about it. If nothing >> replicates, that's good to know. If everything replicates, that's >> good to know. It's only positive because it means we know more and, if >> need be, can make improvements. We are not planning to have a >> witch-hunt at the end of this--going after researchers whose studies >> didn't replicate--we're going to seek collaboration with them (as we >> already have been doing) in order to dig deeper into the substantive >> problem space and figure out why there are discrepancies.
>> Jeff.
>> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> wrote: >> > I also don't know how much negative press will affect funding >> decisions. Hopefully it won't, but I think we can do simple things >> like present comparison tables to try to prevent bad outcomes, not >> matter how unlikely. Maybe I'm particularly sensitive because in my >> state (Alabama), medical doctors (psychiatrists) are trying to >> remove the terms "diagnose" and' "treat" from the official job >> description of psychologists. They essentially argue that >> psychology lacks scientific credibility and that psychologists >> should not be entrusted to do these types of activities--that these >> activities should be left to the "real doctors." If we're going to >> present them a low replication rate for psychology, I want to at >> least make sure they see medicine's replication rate right next to >> it.
>> > ______________________________ >> > Joshua D. Foster, Ph.D. >> > Associate Professor >> > Psychology Department >> > University of South Alabama >> > Mobile, AL 36688 >> > (251) 460-6548 >> >http://www.southalabama.edu/psychology/Faculty_Foster.htm
>> >>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>> >> > As somewhat of an outsider to the replication project, my view is >> that it is best not to get too involved with the press on these >> issues, the incentives are too misaligned. We want the best science >> in the long term, they want sensationalist stories that will sell >> papers tomorrow. We care about the reputations of the individuals >> being replicated and of those doing the replicating, they do not >> care about reputations at all.
>> > I agree with EJ that nobody will remember the stories within 48 >> hours, and I am a bit skeptical that such stories really have much >> of an impact on funding (though little evidence behind that so >> maybe just being optimistic).
>> > I worry quite a bit, however, about how the popular press >> distorts the dialogue within psychology.
>> > So take a psychologist who has not thought too much about >> replications, methodology, false-positive, fabrication, etc. (i.e., >> the modal scientist) And that psychologist reads about replication >> attempts (or fabrication or whatever) in the Times, if the tone is >> not perfect, if she feels her field is under attack, she becomes >> negatively predisposed to the endeveour of open science, >> replication, skepticism, etc, and she does NOT forget about it >> within 72 hours. She takes it with her to conference discussion, >> referee reports, decisions of whom to invite for talks, etc.
>> > The argument, I guess, is that if one wants to be persuasive >> within the field, involving the press has the downside of being >> potentially counterproductive and the upside is not obvious.
>> > My 2 cents, again, as mostly an outsider, I may be missing a lot >> of perspective on this,
>> > Uri
>> > From: openscienceframework@googlegroups.com >> [mailto:openscienceframework@googlegroups.com] On Behalf Of Frank >> Farach >> > Sent: Wednesday, April 18, 2012 12:20 PM >> > To: openscienceframework@googlegroups.com >> > Subject: Re: [OpenScienceFramework] Chronicle coverage
>> > The idea of benchmarking our replication metrics against other >> systematic replication projects is intriguing and might provide >> valuable context to the discussion, as Joshua mentioned. Is anyone >> aware of such efforts outside of psychology?
>> > Negative press doesn't bother me much as long as it doesn't gain >> traction in the politics of funding for psychological science.
>> > Frank
>> > -- >> > Frank J. Farach, Ph.D. >> > Postdoctoral Fellow >> > Center for Anxiety and Traumatic Stress >> > University of Washington >> > Department of Psychology >> > Guthrie Hall, Box 351525 >> > Seattle, WA 98195-1525
>> > On Wed, Apr 18, 2012 at 7:58 AM, Joshua Foster >> <fos...@usouthal.edu<mailto:fos...@usouthal.edu>> wrote: >> > I think it is reasonable to be concerned about how the results of >> our replication effort will be perceived by people outside of >> psychology. Let's say we find that 40% of our results replicate. >> Personally, I wouldn't consider that a failure of psychology and, >> indeed, it would be fairly consistent with what other sciences seem >> to be finding. But without that context it would be read by most >> "outsiders" as an indictment of psychology. Because of this, I >> think it will be useful to present whatever findings we get >> alongside what
OK, OK, let me play the devil's advocate here (I think it's an important discussion). Suppose we find that replication rates are surprisingly low, despite our best efforts. This reflects poorly on the field as a whole, however way you look at it -- if we cannot replicate what is published, psychology fails the litmus test for scientific credibility. I don't think that this will happen, but let's assume.
So let's say we will establish that psychological findings can hardly be replicated. Tax payers, represented by the government, may now feel that it is wise to tighten the funding for fields in which few findings actually replicate. I realize that it is never pleasant to part with grant money, but wouldn't the tax payers have a point here? As a tax player, how would you see your money allocated?
The defense that tightening funding for psychology is not fair because other fields may *also* have a replicability problem is not compelling, as it only suggest that funding should be cut for other fields as well. We all like to have more money for research, but if we find that psychology has a massive replicability problem it is understandable that we receive less money from the government. Apparently, we do not put the money to good use (again, I don't think this is what our efforts will show, but this is certainly a line of argument that a journalist could consider).
Suppose a police station runs an internal investigation into fraud, and finds that 80% of officers take bribes. The argument that other police stations may are also corrupt is not a good argument against the state taking action for that particular police station.
So I see the pragmatics (i.e., we don't want to lose funding because we are being self-critical, which is a good thing), but I am just not sure whether the fear of seeing funding evaporate carries much weight in the end.
I guess it is possible to argue the other way around as well: if psychology is in a bad state we need more money to clean up our act.
On Thu, Apr 19, 2012 at 5:27 PM, Robert Smith <robw...@umich.edu> wrote: > Great points. This is probably the most coherent thing I've read on this > message board.
>> I think that there is a legitimate concern that the perception and >> affective tone that the findings leave with decision makers and >> citizens will not correspond with the conclusions that we draw from >> the findings. What is said needs to be thought through carefully, with >> an eye to how people on the outside of the field, and inside it may >> see it.
>> First, regardless of what we find, whether a problem exists or not is >> a comparative question and the standard against which the success rate >> is judged is malleable. While we may believe that any replication rate >> that is close to 80% is ok, the finding can still be framed as "more >> than 20% of findings are false." This may not be a deliberate >> misrepresentation, many people have difficulty with the idea of "80% >> accurate science" because science is perceived to be absolute. In >> other words, it isn't obvious that there is social consensus on what >> rate we "should" replicate at.
>> In a similar comparative vein, we all seem to assume that if we as a >> field move first, people will compare our efforts to that of other >> fields and conclude "they must be progressive for doing this" but for >> all we know, people could also infer "there is vivid evidence for non- >> replicability in psych, yet I can't recall any evidence of problems >> with econ, sociology or biology. Psych must have a more serious >> problem than they do." This is not a reason not to do this project, >> but the concern that this perception could emerge is legitimate and >> needs to be considered carefully.
>> I also agree with Uri that the press is not the proper venue for these >> discussions, especially without data. Everything turning out ok is not >> a good story, and reporters are incentivized to find another angle. >> There are too many degrees of freedom within which the tale can be >> told. This may lead to yet more damage, either directly, or indirectly >> thorough researchers interpretation of this project as presented by >> the press. Note in the recent piece that by merely associating this >> project with Stapel and Bem, it could easily leave one with the >> impression that these exemplars are representative of the studies that >> fail to replicate. In other words, 80% of us are Stapels. One that >> becomes the remembered message, it is not clear how to unsay it.
>> One thing that also requires some thought is the extent to which we >> want to make the individual failures to replicate explicit. One option >> might be to present only the aggregate data in the actual writeup, >> with specific failures to replicate in the supplementary materials for >> those who care (I am not interested in the opinions of people to lazy >> to read the supplemental materials anyway). At the same time, I also >> believe that refusal to share materials should be made explicit >> because this is contrary to the core values of science.
>> Jesse
>> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote:
>>> I have just a few comments to add to this discussion:
>>> 1) Why are the people who are critical about the project so concerned >>> that this is going to destroy psychology? It assumes that we won't >>> replicate at the rate we should. But why would that be the case? If >>> there wasn't a problem in the field, why would they be worried about >>> this in the first place? Intrinsic in their doubt is an assumption >>> that a problem exists! Perhaps they worry that we're bad scientists >>> or we're going to make stuff up, but why is that specific to this >>> group? I have a feeling that that's not what they're worried about.
>>> 2) We should continue communicating our work in an open, honest, >>> scientific way. Should we pay special attention to making the work >>> understandable even by a lay audience? Absolutely; but we always >>> should. This just happens to be a case where a lay audience is likely >>> watching more closely.
>>> 3) In the end, as most of you probably agree, there are ONLY positives >>> coming out of this project. We should praise psychology for being so >>> progressive that a group of scientists like those in the Open Science >>> Framework/Collaboration are willing to embark on such a project that >>> sacrifices personal gain for the good of science. We should praise >>> psychology that there is this esteem for such a vital piece of the >>> scientific process. And we should praise psychology for recognizing >>> the problems both broadly in science and in our field and not just >>> talking about it--or worse--not NOT talking about it. If nothing >>> replicates, that's good to know. If everything replicates, that's >>> good to know. It's only positive because it means we know more and, if >>> need be, can make improvements. We are not planning to have a >>> witch-hunt at the end of this--going after researchers whose studies >>> didn't replicate--we're going to seek collaboration with them (as we >>> already have been doing) in order to dig deeper into the substantive >>> problem space and figure out why there are discrepancies.
>>> Jeff.
>>> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> >>> wrote: >>> > I also don't know how much negative press will affect funding >>> > decisions. Hopefully it won't, but I think we can do simple things like >>> > present comparison tables to try to prevent bad outcomes, not matter how >>> > unlikely. Maybe I'm particularly sensitive because in my state (Alabama), >>> > medical doctors (psychiatrists) are trying to remove the terms "diagnose" >>> > and' "treat" from the official job description of psychologists. They >>> > essentially argue that psychology lacks scientific credibility and that >>> > psychologists should not be entrusted to do these types of activities--that >>> > these activities should be left to the "real doctors." If we're going to >>> > present them a low replication rate for psychology, I want to at least make >>> > sure they see medicine's replication rate right next to it.
>>> > ______________________________ >>> > Joshua D. Foster, Ph.D. >>> > Associate Professor >>> > Psychology Department >>> > University of South Alabama >>> > Mobile, AL 36688 >>> > (251) 460-6548 >>> >http://www.southalabama.edu/psychology/Faculty_Foster.htm
>>> >>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>> >>> > As somewhat of an outsider to the replication project, my view is that >>> > it is best not to get too involved with the press on these issues, the >>> > incentives are too misaligned. We want the best science in the long term, >>> > they want sensationalist stories that will sell papers tomorrow. We care >>> > about the reputations of the individuals being replicated and of those doing >>> > the replicating, they do not care about reputations at all.
>>> > I agree with EJ that nobody will remember the stories within 48 hours, >>> > and I am a bit skeptical that such stories really have much of an impact on >>> > funding (though little evidence behind that so maybe just being optimistic).
>>> > I worry quite a bit, however, about how the popular press distorts the >>> > dialogue within psychology.
>>> > So take a psychologist who has not thought too much about replications, >>> > methodology, false-positive, fabrication, etc. (i.e., the modal scientist) >>> > And that psychologist reads about replication attempts (or fabrication or >>> > whatever) in the Times, if the tone is not perfect, if she feels her field >>> > is under attack, she becomes negatively predisposed to the endeveour of open >>> > science, replication, skepticism, etc, and she
> OK, OK, let me play the devil's advocate here (I think it's an > important discussion). Suppose we find that replication rates are > surprisingly low, despite our best efforts. This reflects poorly on > the field as a whole, however way you look at it -- if we cannot > replicate what is published, psychology fails the litmus test for > scientific credibility. I don't think that this will happen, but let's > assume.
> So let's say we will establish that psychological findings can hardly > be replicated. Tax payers, represented by the government, may now feel > that it is wise to tighten the funding for fields in which few > findings actually replicate. I realize that it is never pleasant to > part with grant money, but wouldn't the tax payers have a point here? > As a tax player, how would you see your money allocated?
> The defense that tightening funding for psychology is not fair because > other fields may *also* have a replicability problem is not > compelling, as it only suggest that funding should be cut for other > fields as well. We all like to have more money for research, but if we > find that psychology has a massive replicability problem it is > understandable that we receive less money from the government. > Apparently, we do not put the money to good use (again, I don't think > this is what our efforts will show, but this is certainly a line of > argument that a journalist could consider).
> Suppose a police station runs an internal investigation into fraud, > and finds that 80% of officers take bribes. The argument that other > police stations may are also corrupt is not a good argument against > the state taking action for that particular police station.
> So I see the pragmatics (i.e., we don't want to lose funding because > we are being self-critical, which is a good thing), but I am just not > sure whether the fear of seeing funding evaporate carries much weight > in the end.
> I guess it is possible to argue the other way around as well: if > psychology is in a bad state we need more money to clean up our act.
> Cheers, > E.J.
> On Thu, Apr 19, 2012 at 5:27 PM, Robert Smith <robw...@umich.edu> wrote: >> Great points. This is probably the most coherent thing I've read on this >> message board.
>>> I think that there is a legitimate concern that the perception and >>> affective tone that the findings leave with decision makers and >>> citizens will not correspond with the conclusions that we draw from >>> the findings. What is said needs to be thought through carefully, with >>> an eye to how people on the outside of the field, and inside it may >>> see it.
>>> First, regardless of what we find, whether a problem exists or not is >>> a comparative question and the standard against which the success rate >>> is judged is malleable. While we may believe that any replication rate >>> that is close to 80% is ok, the finding can still be framed as "more >>> than 20% of findings are false." This may not be a deliberate >>> misrepresentation, many people have difficulty with the idea of "80% >>> accurate science" because science is perceived to be absolute. In >>> other words, it isn't obvious that there is social consensus on what >>> rate we "should" replicate at.
>>> In a similar comparative vein, we all seem to assume that if we as a >>> field move first, people will compare our efforts to that of other >>> fields and conclude "they must be progressive for doing this" but for >>> all we know, people could also infer "there is vivid evidence for non- >>> replicability in psych, yet I can't recall any evidence of problems >>> with econ, sociology or biology. Psych must have a more serious >>> problem than they do." This is not a reason not to do this project, >>> but the concern that this perception could emerge is legitimate and >>> needs to be considered carefully.
>>> I also agree with Uri that the press is not the proper venue for these >>> discussions, especially without data. Everything turning out ok is not >>> a good story, and reporters are incentivized to find another angle. >>> There are too many degrees of freedom within which the tale can be >>> told. This may lead to yet more damage, either directly, or indirectly >>> thorough researchers interpretation of this project as presented by >>> the press. Note in the recent piece that by merely associating this >>> project with Stapel and Bem, it could easily leave one with the >>> impression that these exemplars are representative of the studies that >>> fail to replicate. In other words, 80% of us are Stapels. One that >>> becomes the remembered message, it is not clear how to unsay it.
>>> One thing that also requires some thought is the extent to which we >>> want to make the individual failures to replicate explicit. One option >>> might be to present only the aggregate data in the actual writeup, >>> with specific failures to replicate in the supplementary materials for >>> those who care (I am not interested in the opinions of people to lazy >>> to read the supplemental materials anyway). At the same time, I also >>> believe that refusal to share materials should be made explicit >>> because this is contrary to the core values of science.
>>> Jesse
>>> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote:
>>>> I have just a few comments to add to this discussion:
>>>> 1) Why are the people who are critical about the project so concerned >>>> that this is going to destroy psychology? It assumes that we won't >>>> replicate at the rate we should. But why would that be the case? If >>>> there wasn't a problem in the field, why would they be worried about >>>> this in the first place? Intrinsic in their doubt is an assumption >>>> that a problem exists! Perhaps they worry that we're bad scientists >>>> or we're going to make stuff up, but why is that specific to this >>>> group? I have a feeling that that's not what they're worried about.
>>>> 2) We should continue communicating our work in an open, honest, >>>> scientific way. Should we pay special attention to making the work >>>> understandable even by a lay audience? Absolutely; but we always >>>> should. This just happens to be a case where a lay audience is likely >>>> watching more closely.
>>>> 3) In the end, as most of you probably agree, there are ONLY positives >>>> coming out of this project. We should praise psychology for being so >>>> progressive that a group of scientists like those in the Open Science >>>> Framework/Collaboration are willing to embark on such a project that >>>> sacrifices personal gain for the good of science. We should praise >>>> psychology that there is this esteem for such a vital piece of the >>>> scientific process. And we should praise psychology for recognizing >>>> the problems both broadly in science and in our field and not just >>>> talking about it--or worse--not NOT talking about it. If nothing >>>> replicates, that's good to know. If everything replicates, that's >>>> good to know. It's only positive because it means we know more and, if >>>> need be, can make improvements. We are not planning to have a >>>> witch-hunt at the end of this--going after researchers whose studies >>>> didn't replicate--we're going to seek collaboration with them (as we >>>> already have been doing) in order to dig deeper into the substantive >>>> problem space and figure out why there are discrepancies.
>>>> Jeff.
>>>> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> >>>> wrote: >>>>> I also don't know how much negative press will affect funding >>>>> decisions. Hopefully it won't, but I think we can do simple things like >>>>> present comparison tables to try to prevent bad outcomes, not matter how >>>>> unlikely. Maybe I'm particularly sensitive because in my state (Alabama), >>>>> medical doctors (psychiatrists) are trying to remove the terms "diagnose" >>>>> and' "treat" from the official job description of psychologists. They >>>>> essentially argue that psychology lacks scientific credibility and that >>>>> psychologists should not be entrusted to do these types of activities--that >>>>> these activities should be left to the "real doctors." If we're going to >>>>> present them a low replication rate for psychology, I want to at least make >>>>> sure they see medicine's replication rate right next to it.
>>>>> ______________________________ >>>>> Joshua D. Foster, Ph.D. >>>>> Associate Professor >>>>> Psychology Department >>>>> University of South Alabama >>>>> Mobile, AL 36688 >>>>> (251) 460-6548 >>>>> http://www.southalabama.edu/psychology/Faculty_Foster.htm
>>>>>>>> "Simonsohn, Uri" <u...@wharton.upenn.edu> 4/18/2012 12:40 PM >>> >>>>> As somewhat of an outsider to the replication project, my view is that >>>>> it is best not to get too involved with the press on these issues, the >>>>> incentives are too misaligned. We want the best science in the long term, >>>>> they want sensationalist stories that will sell papers tomorrow. We care >>>>> about the reputations of the individuals being replicated and of those doing >>>>> the replicating, they do not care about reputations at all.
>>>>> I agree with EJ that nobody will remember the stories within 48 hours, >>>>> and I am a bit skeptical that such stories really have much of an impact on >>>>> funding (though little evidence behind that so maybe just being
If the folks who control the purse string were as bright and thoughtful as you, E.J., then I wouldn't be worried about any of this. I agree with everything you said, but especially the last sentence. People may dismiss me as a pessimist, but I think psychologists are trying to do the near impossible--that is, try to measure and manipulate the most complicated thing in the known universe with crude instruments. If we're right even some of the time, then in my mind that is a huge accomplishment and worthy of continued effort.
______________________________ Joshua D. Foster, Ph.D. Associate Professor Psychology Department University of South Alabama Mobile, AL 36688 (251) 460-6548 www.joshuadfoster.com
OK, OK, let me play the devil's advocate here (I think it's an important discussion). Suppose we find that replication rates are surprisingly low, despite our best efforts. This reflects poorly on the field as a whole, however way you look at it -- if we cannot replicate what is published, psychology fails the litmus test for scientific credibility. I don't think that this will happen, but let's assume.
So let's say we will establish that psychological findings can hardly be replicated. Tax payers, represented by the government, may now feel that it is wise to tighten the funding for fields in which few findings actually replicate. I realize that it is never pleasant to part with grant money, but wouldn't the tax payers have a point here? As a tax player, how would you see your money allocated?
The defense that tightening funding for psychology is not fair because other fields may *also* have a replicability problem is not compelling, as it only suggest that funding should be cut for other fields as well. We all like to have more money for research, but if we find that psychology has a massive replicability problem it is understandable that we receive less money from the government. Apparently, we do not put the money to good use (again, I don't think this is what our efforts will show, but this is certainly a line of argument that a journalist could consider).
Suppose a police station runs an internal investigation into fraud, and finds that 80% of officers take bribes. The argument that other police stations may are also corrupt is not a good argument against the state taking action for that particular police station.
So I see the pragmatics (i.e., we don't want to lose funding because we are being self-critical, which is a good thing), but I am just not sure whether the fear of seeing funding evaporate carries much weight in the end.
I guess it is possible to argue the other way around as well: if psychology is in a bad state we need more money to clean up our act.
On Thu, Apr 19, 2012 at 5:27 PM, Robert Smith <robw...@umich.edu> wrote: > Great points. This is probably the most coherent thing I've read on this > message board.
>> I think that there is a legitimate concern that the perception and >> affective tone that the findings leave with decision makers and >> citizens will not correspond with the conclusions that we draw from >> the findings. What is said needs to be thought through carefully, with >> an eye to how people on the outside of the field, and inside it may >> see it.
>> First, regardless of what we find, whether a problem exists or not is >> a comparative question and the standard against which the success rate >> is judged is malleable. While we may believe that any replication rate >> that is close to 80% is ok, the finding can still be framed as "more >> than 20% of findings are false." This may not be a deliberate >> misrepresentation, many people have difficulty with the idea of "80% >> accurate science" because science is perceived to be absolute. In >> other words, it isn't obvious that there is social consensus on what >> rate we "should" replic>> field move first, people will compare our efforts to that of other >> fields and conclude "they must be progressive for doing this" but for >> all we know, people could also infer "there is vivid evidence for non- >> replicability in psych, yet I can't recall any evidence of problems >> with econ, sociology or biology. Psych must have a more serious >> problem than they do." This is not a reason not to do this project, >> but the concern that this perception could emerge is legitimate and >> needs to be considered carefully.
>> I also agree with Uri that the press is not the proper venue for these >> discussions, especially without data. Everything turning out ok is not >> a good story, and reporters are incentivized to find another angle. >> There are too many degrees of freedom within which the tale can be >> told. This may lead to yet more damage, either directly, or indirectly >> thorough researchers interpretation of this project as presented by >> the press. Note in the recent piece that by merely associating this >> project with Stapel and Bem, it could easily leave one with the >> impression that these exemplars are representative of the studies that >> fail to replicate. In other words, 80% of us are Stapels. One that >> becomes the remembered message, it is not clear how to unsay it.
>> One thing that also requires some thought is the extent to which we >> want to make the individual failures to replicate explicit. One option >> might be to present only the aggregate data in the actual writeup, >> with specific failures to replicate in the supplementary materials for >> those who care (I am not interested in the opinions of people to lazy >> to read the supplemental materials anyway). At the same time, I also >> believe that refusal to share materials should be made explicit >> because this is contrary to the core values of science.
>> Jesse
>> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote:
>>> I have just a few comments to add to this discussion:
>>> 1) Why are the people who are critical about the project so concerned >>> that this is going to destroy psychology? It assumes that we won't >>> replicate at the rate we should. But why would that be the case? If >>> there wasn't a problem in the field, why would they be worried about >>> this in the first place? Intrinsic in their doubt is an assumption >>> that a problem exists! Perhaps they worry that we're bad scientists >>> or we're going to make stuff up, but why is that specific to this >>> group? I have a feeling that that's not what they're worried about.
>>> 2) We should continue communicating our work in an open, honest, >>> scientific way. Should we pay special attention to making the work >>> understandable even by a lay audience? Absolutely; but we always >>> should. This just happens to be a case where a lay audience is likely >>> watching more closely.
>>> 3) In the end, as most of you probably agree, there are ONLY positives >>> coming out of this project. We should praise psychology for being so >>> progressive that a group of scientists like those in the Open Science >>> Framework/Collaboration are willing to embark on such a project that >>> sacrifices personal gain for the good of science. We should praise >>> psychology that there is this esteem for such a vital piece of the >>> scientific process. And we should praise psychology for recognizing >>> the problems both broadly in science and in our field and not just >>> talking about it--or worse--not NOT talking about it. If nothing >>> replicates, that's good to know. If everything replicates, that's >>> good to know. It's only positive because it means we know more and, if >>> need be, can make improvements. We are not planning to have a >>> witch-hunt at the end of this--going after researchers whose studies >>> didn't replicate--we're going to seek collaboration with them (as we >>> already have been doing) in order to dig deeper into the substantive
>>> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> >>> wrote: >>> > I also don't know how much negative press will affect funding >>> > decisions. Hopefully it won't, but I think we can do simple things like >>> > present comparison tables to try to prevent bad outcomes, not matter how >>> > unlikely. Maybe I'm particularly sensitive because in my state (Alabama), >>> > medical doctors (psychiatrists) are trying to remove the terms "diagnose" >>> > and' "treat" from the official job description of psychologists. They >>> > essentially argue that psychology lacks scientific credibility and that >>> > psychologists should not be entrusted to do these types of activities--that >>> > these activities should be left to the "real doctors." If we're going to >>> > present them a low replication rate for psychology, I want to at least make >>> > sure they see medicine's replication rate right next to it.
>>> > ______________________________ >>> > Joshua D. Foster, Ph.D. >>> > Associate Professor >>> > Psychology Department >>> > University of South Alabama >>> > Mobile, AL 36688 >>> > (251) 460-6548 >>> >http://www.southalabama.edu/psychology/Faculty_Foster.htm
>>> > As somewhat of an outsider to the replication project, my view is that >>> > it is best not to get too involved with the press on these issues, the >>> > incentives are too misaligned. We want the best science in the long term, >>> > they want sensationalist stories that will sell papers tomorrow. We care >>> > about the reputations of the individuals being replicated and of those doing >>> > the replicating, they do not care about reputations at all.
>>> > I agree with EJ that nobody will remember the stories within 48 hours, >>> > and I am a bit skeptical that such stories really have much of an impact on >>> > funding (though little evidence behind that so
Interestingly, some have claimed that simple induction from the past (which they argue shows that even best theories eventually fall) leads to the conclusion that all present theories are false as well. The idea is known as pessimistic induction.
Although I don't personally buy the argument, I do think that a realistic assessment of the state of affairs would suggest that most of what we believe is probably wrong, in psychology and elsewhere.
Science is not a tea party. It is hard and unforgiving and it will kill all your bedtime stories if your wait long enough. But it is the most successful epistemic enterprise in the history of mankind, and I am happy and proud to be a part of it.
There will certainly be people who try to misuse results any investigation to their financial benefit, and the current study is no exception. However, there are also many people (some of the with their hands on a purse string) who understand the basic idea of science and who will act to reward the movement towards openness that is currently taking place.
On Fri, Apr 20, 2012 at 5:50 AM, Joshua Foster <fos...@usouthal.edu> wrote: > If the folks who control the purse string were as bright and thoughtful > as you, E.J., then I wouldn't be worried about any of this. I agree with > everything you said, but especially the last sentence. People may > dismiss me as a pessimist, but I think psychologists are trying to do > the near impossible--that is, try to measure and manipulate the most > complicated thing in the known universe with crude instruments. If we're > right even some of the time, then in my mind that is a huge > accomplishment and worthy of continued effort.
> ______________________________ > Joshua D. Foster, Ph.D. > Associate Professor > Psychology Department > University of South Alabama > Mobile, AL 36688 > (251) 460-6548 > www.joshuadfoster.com >>>> Eric-Jan Wagenmakers <ej.wagenmak...@gmail.com> 04/19/12 3:09 PM >>> > OK, OK, let me play the devil's advocate here (I think it's an > important discussion). Suppose we find that replication rates are > surprisingly low, despite our best efforts. This reflects poorly on > the field as a whole, however way you look at it -- if we cannot > replicate what is published, psychology fails the litmus test for > scientific credibility. I don't think that this will happen, but let's > assume.
> So let's say we will establish that psychological findings can hardly > be replicated. Tax payers, represented by the government, may now feel > that it is wise to tighten the funding for fields in which few > findings actually replicate. I realize that it is never pleasant to > part with grant money, but wouldn't the tax payers have a point here? > As a tax player, how would you see your money allocated?
> The defense that tightening funding for psychology is not fair because > other fields may *also* have a replicability problem is not > compelling, as it only suggest that funding should be cut for other > fields as well. We all like to have more money for research, but if we > find that psychology has a massive replicability problem it is > understandable that we receive less money from the government. > Apparently, we do not put the money to good use (again, I don't think > this is what our efforts will show, but this is certainly a line of > argument that a journalist could consider).
> Suppose a police station runs an internal investigation into fraud, > and finds that 80% of officers take bribes. The argument that other > police stations may are also corrupt is not a good argument against > the state taking action for that particular police station.
> So I see the pragmatics (i.e., we don't want to lose funding because > we are being self-critical, which is a good thing), but I am just not > sure whether the fear of seeing funding evaporate carries much weight > in the end.
> I guess it is possible to argue the other way around as well: if > psychology is in a bad state we need more money to clean up our act.
> Cheers, > E.J.
> On Thu, Apr 19, 2012 at 5:27 PM, Robert Smith <robw...@umich.edu> wrote: >> Great points. This is probably the most coherent thing I've read on > this >> message board.
>>> I think that there is a legitimate concern that the perception and >>> affective tone that the findings leave with decision makers and >>> citizens will not correspond with the conclusions that we draw from >>> the findings. What is said needs to be thought through carefully, > with >>> an eye to how people on the outside of the field, and inside it may >>> see it.
>>> First, regardless of what we find, whether a problem exists or not is >>> a comparative question and the standard against which the success > rate >>> is judged is malleable. While we may believe that any replication > rate >>> that is close to 80% is ok, the finding can still be framed as "more >>> than 20% of findings are false." This may not be a deliberate >>> misrepresentation, many people have difficulty with the idea of "80% >>> accurate science" because science is perceived to be absolute. In >>> other words, it isn't obvious that there is social consensus on what >>> rate we "should" replic>> field move first, people will compare our efforts to that of other >>> fields and conclude "they must be progressive for doing this" but for >>> all we know, people could also infer "there is vivid evidence for > non- >>> replicability in psych, yet I can't recall any evidence of problems >>> with econ, sociology or biology. Psych must have a more serious >>> problem than they do." This is not a reason not to do this project, >>> but the concern that this perception could emerge is legitimate and >>> needs to be considered carefully.
>>> I also agree with Uri that the press is not the proper venue for > these >>> discussions, especially without data. Everything turning out ok is > not >>> a good story, and reporters are incentivized to find another angle. >>> There are too many degrees of freedom within which the tale can be >>> told. This may lead to yet more damage, either directly, or > indirectly >>> thorough researchers interpretation of this project as presented by >>> the press. Note in the recent piece that by merely associating this >>> project with Stapel and Bem, it could easily leave one with the >>> impression that these exemplars are representative of the studies > that >>> fail to replicate. In other words, 80% of us are Stapels. One that >>> becomes the remembered message, it is not clear how to unsay it.
>>> One thing that also requires some thought is the extent to which we >>> want to make the individual failures to replicate explicit. One > option >>> might be to present only the aggregate data in the actual writeup, >>> with specific failures to replicate in the supplementary materials > for >>> those who care (I am not interested in the opinions of people to lazy >>> to read the supplemental materials anyway). At the same time, I also >>> believe that refusal to share materials should be made explicit >>> because this is contrary to the core values of science.
>>> Jesse
>>> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote:
>>>> I have just a few comments to add to this discussion:
>>>> 1) Why are the people who are critical about the project so > concerned >>>> that this is going to destroy psychology? It assumes that we won't >>>> replicate at the rate we should. But why would that be the case? If >>>> there wasn't a problem in the field, why would they be worried about >>>> this in the first place? Intrinsic in their doubt is an assumption >>>> that a problem exists! Perhaps they worry that we're bad scientists >>>> or we're going to make stuff up, but why is that specific to this >>>> group? I have a feeling that that's not what they're worried about.
>>>> 2) We should continue communicating our work in an open, honest, >>>> scientific way. Should we pay special attention to making the work >>>> understandable even by a lay audience? Absolutely; but we always >>>> should. This just happens to be a case where a lay audience is > likely >>>> watching more closely.
>>>> 3) In the end, as most of you probably agree, there are ONLY > positives >>>> coming out of this project. We should praise psychology for being > so >>>> progressive that a group of scientists like those in the Open > Science >>>> Framework/Collaboration are willing to embark on such a project that >>>> sacrifices personal gain for the good of science. We should praise >>>> psychology that there is this esteem for such a vital piece of the >>>> scientific process. And we should praise psychology for recognizing >>>> the problems both broadly in science and in our field and not just >>>> talking about it--or worse--not NOT talking about it. If nothing >>>> replicates, that's good to know. If everything replicates, that's >>>> good to know. It's only positive because it means we know more and, > if >>>> need be, can make improvements. We are not planning to have a >>>> witch-hunt at the end of this--going after researchers whose studies >>>> didn't replicate--we're going to seek collaboration with them (as we >>>> already have been doing) in order to dig deeper into the substantive
>>>> On Wed, Apr 18, 2012 at 1:57 PM, Joshua Foster <fos...@usouthal.edu> >>>> wrote: >>>> > I also don't know how much negative press will affect funding >>>> > decisions. Hopefully it won't, but I think we can do simple things > like >>>> > present comparison tables to try to prevent bad outcomes, not > matter how >>>> > unlikely. Maybe I'm particularly sensitive because in my state > (Alabama), >>>> > medical doctors (psychiatrists) are trying to remove the terms > "diagnose" >>>> > and' "treat"
Hi Denny Even better, that inference is sometimes called "The Pessimistic Meta-Induction from the History of Science" - one of my favourite technical terms. Rebecca
On Apr 20, 2012, at 4:32 AM, Denny Borsboom wrote:
> Interestingly, some have claimed that simple induction from the past > (which they argue shows that even best theories eventually fall) leads > to the conclusion that all present theories are false as well. The > idea is known as pessimistic induction.
> Although I don't personally buy the argument, I do think that a > realistic assessment of the state of affairs would suggest that most > of what we believe is probably wrong, in psychology and elsewhere.
> Science is not a tea party. It is hard and unforgiving and it will > kill all your bedtime stories if your wait long enough. But it is the > most successful epistemic enterprise in the history of mankind, and I > am happy and proud to be a part of it.
> There will certainly be people who try to misuse results any > investigation to their financial benefit, and the current study is no > exception. However, there are also many people (some of the with their > hands on a purse string) who understand the basic idea of science and > who will act to reward the movement towards openness that is currently > taking place.
> Best > Denny
> On Fri, Apr 20, 2012 at 5:50 AM, Joshua Foster <fos...@usouthal.edu> wrote: >> If the folks who control the purse string were as bright and thoughtful >> as you, E.J., then I wouldn't be worried about any of this. I agree with >> everything you said, but especially the last sentence. People may >> dismiss me as a pessimist, but I think psychologists are trying to do >> the near impossible--that is, try to measure and manipulate the most >> complicated thing in the known universe with crude instruments. If we're >> right even some of the time, then in my mind that is a huge >> accomplishment and worthy of continued effort.
>> ______________________________ >> Joshua D. Foster, Ph.D. >> Associate Professor >> Psychology Department >> University of South Alabama >> Mobile, AL 36688 >> (251) 460-6548 >> www.joshuadfoster.com >>>>> Eric-Jan Wagenmakers <ej.wagenmak...@gmail.com> 04/19/12 3:09 PM >>> >> OK, OK, let me play the devil's advocate here (I think it's an >> important discussion). Suppose we find that replication rates are >> surprisingly low, despite our best efforts. This reflects poorly on >> the field as a whole, however way you look at it -- if we cannot >> replicate what is published, psychology fails the litmus test for >> scientific credibility. I don't think that this will happen, but let's >> assume.
>> So let's say we will establish that psychological findings can hardly >> be replicated. Tax payers, represented by the government, may now feel >> that it is wise to tighten the funding for fields in which few >> findings actually replicate. I realize that it is never pleasant to >> part with grant money, but wouldn't the tax payers have a point here? >> As a tax player, how would you see your money allocated?
>> The defense that tightening funding for psychology is not fair because >> other fields may *also* have a replicability problem is not >> compelling, as it only suggest that funding should be cut for other >> fields as well. We all like to have more money for research, but if we >> find that psychology has a massive replicability problem it is >> understandable that we receive less money from the government. >> Apparently, we do not put the money to good use (again, I don't think >> this is what our efforts will show, but this is certainly a line of >> argument that a journalist could consider).
>> Suppose a police station runs an internal investigation into fraud, >> and finds that 80% of officers take bribes. The argument that other >> police stations may are also corrupt is not a good argument against >> the state taking action for that particular police station.
>> So I see the pragmatics (i.e., we don't want to lose funding because >> we are being self-critical, which is a good thing), but I am just not >> sure whether the fear of seeing funding evaporate carries much weight >> in the end.
>> I guess it is possible to argue the other way around as well: if >> psychology is in a bad state we need more money to clean up our act.
>> Cheers, >> E.J.
>> On Thu, Apr 19, 2012 at 5:27 PM, Robert Smith <robw...@umich.edu> wrote: >>> Great points. This is probably the most coherent thing I've read on >> this >>> message board.
>>>> I think that there is a legitimate concern that the perception and >>>> affective tone that the findings leave with decision makers and >>>> citizens will not correspond with the conclusions that we draw from >>>> the findings. What is said needs to be thought through carefully, >> with >>>> an eye to how people on the outside of the field, and inside it may >>>> see it.
>>>> First, regardless of what we find, whether a problem exists or not is >>>> a comparative question and the standard against which the success >> rate >>>> is judged is malleable. While we may believe that any replication >> rate >>>> that is close to 80% is ok, the finding can still be framed as "more >>>> than 20% of findings are false." This may not be a deliberate >>>> misrepresentation, many people have difficulty with the idea of "80% >>>> accurate science" because science is perceived to be absolute. In >>>> other words, it isn't obvious that there is social consensus on what >>>> rate we "should" replic>> field move first, people will compare our efforts to that of other >>>> fields and conclude "they must be progressive for doing this" but for >>>> all we know, people could also infer "there is vivid evidence for >> non- >>>> replicability in psych, yet I can't recall any evidence of problems >>>> with econ, sociology or biology. Psych must have a more serious >>>> problem than they do." This is not a reason not to do this project, >>>> but the concern that this perception could emerge is legitimate and >>>> needs to be considered carefully.
>>>> I also agree with Uri that the press is not the proper venue for >> these >>>> discussions, especially without data. Everything turning out ok is >> not >>>> a good story, and reporters are incentivized to find another angle. >>>> There are too many degrees of freedom within which the tale can be >>>> told. This may lead to yet more damage, either directly, or >> indirectly >>>> thorough researchers interpretation of this project as presented by >>>> the press. Note in the recent piece that by merely associating this >>>> project with Stapel and Bem, it could easily leave one with the >>>> impression that these exemplars are representative of the studies >> that >>>> fail to replicate. In other words, 80% of us are Stapels. One that >>>> becomes the remembered message, it is not clear how to unsay it.
>>>> One thing that also requires some thought is the extent to which we >>>> want to make the individual failures to replicate explicit. One >> option >>>> might be to present only the aggregate data in the actual writeup, >>>> with specific failures to replicate in the supplementary materials >> for >>>> those who care (I am not interested in the opinions of people to lazy >>>> to read the supplemental materials anyway). At the same time, I also >>>> believe that refusal to share materials should be made explicit >>>> because this is contrary to the core values of science.
>>>> Jesse
>>>> On Apr 18, 4:34 pm, Jeffrey Spies <jsp...@gmail.com> wrote:
>>>>> I have just a few comments to add to this discussion:
>>>>> 1) Why are the people who are critical about the project so >> concerned >>>>> that this is going to destroy psychology? It assumes that we won't >>>>> replicate at the rate we should. But why would that be the case? If >>>>> there wasn't a problem in the field, why would they be worried about >>>>> this in the first place? Intrinsic in their doubt is an assumption >>>>> that a problem exists! Perhaps they worry that we're bad scientists >>>>> or we're going to make stuff up, but why is that specific to this >>>>> group? I have a feeling that that's not what they're worried about.
>>>>> 2) We should continue communicating our work in an open, honest, >>>>> scientific way. Should we pay special attention to making the work >>>>> understandable even by a lay audience? Absolutely; but we always >>>>> should. This just happens to be a case where a lay audience is >> likely >>>>> watching more closely.
>>>>> 3) In the end, as most of you probably agree, there are ONLY >> positives >>>>> coming out of this project. We should praise psychology for being >> so >>>>> progressive that a group of scientists like those in the Open >> Science >>>>> Framework/Collaboration are willing to embark on such a project that >>>>> sacrifices personal gain for the good of science. We should praise >>>>> psychology that there is this esteem for such a vital piece of the >>>>> scientific process. And we should praise psychology for recognizing >>>>> the problems both broadly in science and in our field and not just >>>>> talking about it--or worse--not NOT talking about it. If nothing >>>>> replicates, that's good to know. If everything replicates, that's >>>>> good to know. It's only positive because it means we know more and, >> if >>>>> need be, can make improvements. We are not planning to have a >>>>> witch-hunt at the end of this--going after researchers whose studies >>>>> didn't replicate--we're going to seek collaboration with them (as we >>>>> already have been doing) in order to dig deeper into the substantive
