Date: Tue, 22 May 2012 02:29:59 -0700 (PDT) From: Roger Giner-Sorolla <rogersebast...@gmail.com> To: openscienceframework@googlegroups.com Message-Id: <db61e1ba-3496-4194-9c55-5ea25d08647b@googlegroups.com> In-Reply-To: <5720d9a2-bf21-488d-a2f0-8714f27717d5@googlegroups.com> References: <39a403c7-9f0e-44c6-a70e-dda3c822fa08@googlegroups.com> <5720d9a2-bf21-488d-a2f0-8714f27717d5@googlegroups.com> Subject: Re: Will Reproducibility Project unearth "an excess of significant findings"? MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_512_25615641.1337678999790" ------=_Part_512_25615641.1337678999790 Content-Type: multipart/alternative; boundary="----=_Part_513_18716715.1337678999791" ------=_Part_513_18716715.1337678999791 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable That's great, Joachim. Are you using the Reproducibility Project sample of= =20 articles specifically? And some power-related questions: what are you doing for studies that=20 report no effect size (all too common in Psych Science), and repeated=20 measures studies that report no correlation among DVs, which is needed for= =20 ES of the repeated measures ANOVA effects? On Monday, May 21, 2012 10:58:57 PM UTC+1, Joachim Vandekerckhove wrote: > > Hi Roger, > > I haven't been active in this group, but I am doing exactly this right=20 > now. An RA is collecting all the relevant statistics now and we plan to= =20 > write a report by the end of the summer. I'm also meeting Greg soon to=20 > discuss the implementation.=20 > I think it would be a worthwhile effort not only in order to provide an= =20 > overview of observed power (post hoc) in the sample, but also to provide= =20 > power-based advice to groups aiming to replicate the studies. I'm happy t= o=20 > communicate more on this if there is an interest. > > Cheers, > Joachim > > > On Monday, May 21, 2012 6:03:42 AM UTC-7, Roger Giner-Sorolla wrote: >> >> Recently, Gregory Francis has had at least two papers applying the=20 >> methods of Ioannidis & Trikalinos to articles in psychology: one looking= at=20 >> Bem's 2011 JPSP precognition article >> >> Francis, G. (2012). Too good to be true: Publication bias in two=20 >> prominent studies from experimental psychology. *Psychonomic Bulletin &= =20 >> Review*, *19*, 1=E2=80=936. >> =20 >> and another looking at the closeness of desirable objects effect from=20 >> Balcetis and Dunning=20 >> >> Francis, G. (2012). The same old New Look: Publication bias in a study o= f=20 >> wishful seeing. *i-Perception*, *3*(3), 176=E2=80=93178. >> >> which elicited a reply and an exchange from the authors (linked from=20 >> http://i-perception.perceptionweb.com/journal/I/volume/3/article/i0519ic= ) >> >> In theory this could go on forever (a search reveals he has another one= =20 >> in press at JEP:General) and of course the "hit list" approach doesn't= =20 >> leave us with very firm grounds for discipline-wide generalizations abou= t=20 >> false-positive bias. My personal reaction is, why go after results one a= t a=20 >> time if this so obviously reflects an endemic practice in the field? >> >> Indeed, it occurred to me that we could do a lot better by looking at ou= r=20 >> sample of the psychology field from the Reproducibility Project. We woul= d=20 >> have to do power analyses for all studies in each articles, of course, b= ut=20 >> we already have the framework in place. Although I have my own archival= =20 >> project I want to start this summer, I'm wondering if anyone could take = on=20 >> organizing this project, or at least give their opinion as to whether it= 's=20 >> worth doing? >> =20 > ------=_Part_513_18716715.1337678999791 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable That's great, Joachim. Are you using the Reproducibility Project sample of = articles specifically?<br><br>And some power-related questions: what are yo= u doing for studies that report no effect size (all too common in Psych Sci= ence), and repeated measures studies that report no correlation among DVs, = which is needed for ES of the repeated measures ANOVA effects?<br><br>On Mo= nday, May 21, 2012 10:58:57 PM UTC+1, Joachim Vandekerckhove wrote:<blockqu= ote class=3D"gmail_quote" style=3D"margin: 0;margin-left: 0.8ex;border-left= : 1px #ccc solid;padding-left: 1ex;">Hi Roger,<div><br></div><div>I haven't= been active in this group, but I am doing exactly this right now. An RA is= collecting all the relevant statistics now and we plan to write a report b= y the end of the summer. I'm also meeting Greg soon to discuss the implemen= tation. </div><div>I think it would be a worthwhile effort not only in= order to provide an overview of observed power (post hoc) in the sample, b= ut also to provide power-based advice to groups aiming to replicate the stu= dies. I'm happy to communicate more on this if there is an interest.</div><= div><br></div><div>Cheers,</div><div>Joachim</div><div><br><br>On Monday, M= ay 21, 2012 6:03:42 AM UTC-7, Roger Giner-Sorolla wrote:<blockquote class= =3D"gmail_quote" style=3D"margin:0;margin-left:0.8ex;border-left:1px #ccc s= olid;padding-left:1ex">Recently, Gregory Franci<font size=3D"2">s has had a= t least two papers applying the methods of Ioannidis & Trikalinos to ar= ticles in psychology: one looking at Bem's 2011 JPSP precognition article<b= r><br></font><div style=3D"line-height:2;padding-left:2em"> <div><font size=3D"2">Francis, G. (2012). Too good to be true: Publicatio= n bias in two prominent studies from experimental psychology. <i>Psychonomi= c Bulletin & Review</i>, <i>19</i>, 1=E2=80=936.</font></div> <font size=3D"2"><span title=3D"url_ver=3DZ39.88-2004&ctx_ver=3DZ39.8= 8-2004&rfr_id=3Dinfo%3Asid%2Fzotero.org%3A2&rft_val_fmt=3Dinfo%3Aof= i%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=3Darticle&rft.atitle=3DToo%= 20good%20to%20be%20true%3A%20Publication%20bias%20in%20two%20prominent%20st= udies%20from%20experimental%20psychology&rft.jtitle=3DPsychonomic%20Bul= letin%20%26%20Review&rft.volume=3D19&rft.aufirst=3DG.&rft.aulas= t=3DFrancis&rft.au=3DG.%20Francis&rft.date=3D2012&rft.pages=3D1= -6&rft.spage=3D1&rft.epage=3D6"> </span></font></div><font size=3D"2"><br>and another looking at the closene= ss of desirable objects effect from Balcetis and Dunning <br><br></font><di= v style=3D"line-height:2;padding-left:2em"><font size=3D"2"> Francis, G. (2012). The same old New Look: Publication bias in a study of= wishful seeing. <i>i-Perception</i>, <i>3</i>(3), 176=E2=80=93178.<br><br> which elicited a reply and an exchange from the authors (linked from=20 <a href=3D"http://i-perception.perceptionweb.com/journal/I/volume/3/article= /i0519ic" target=3D"_blank">http://i-perception.<wbr>perceptionweb.com/jour= nal/I/<wbr>volume/3/article/i0519ic</a>)<br> <br> In theory this could go on forever (a search reveals he has another one=20 in press at JEP:General) and of course the "hit list" approach doesn't leav= e us with very firm grounds for discipline-wide generalizations about false= -positive bias. My personal reaction is, why go after results one at a time= if this so obviously reflects an endemic practice in the field?<br><br>Ind= eed, it occurred to me that we could do a lot=20 better by looking at our sample of the psychology field from the=20 Reproducibility Project. We would have to do power analyses for all=20 studies in each articles, of course, but we already have the framework in place. Although I have my own archival project I want to start this summer= , I'm wondering if anyone could take on organizing this project, or at leas= t give their opinion as to whether it's worth doing?</font><br> <span title=3D"url_ver=3DZ39.88-2004&ctx_ver=3DZ39.88-2004&rfr_id= =3Dinfo%3Asid%2Fzotero.org%3A2&rft_val_fmt=3Dinfo%3Aofi%2Ffmt%3Akev%3Am= tx%3Ajournal&rft.genre=3Darticle&rft.atitle=3DThe%20same%20old%20Ne= w%20Look%3A%20Publication%20bias%20in%20a%20study%20of%20wishful%20seeing&a= mp;rft.jtitle=3Di-Perception&rft.volume=3D3&rft.issue=3D3&rft.a= ufirst=3DG.&rft.aulast=3DFrancis&rft.au=3DG.%20Francis&rft.date= =3D2012&rft.pages=3D176-178&rft.spage=3D176&rft.epage=3D178"> </span></div></blockquote></div></blockquote> ------=_Part_513_18716715.1337678999791-- ------=_Part_512_25615641.1337678999790--
Message from discussion Will Reproducibility Project unearth "an excess of significant findings"?
| Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy |
| ©2013 Google |