idea: "80% and/or 95% power"- badge

[a.]

Badges: we all know them and love them (https://osf.io/tvyxz/).
Power: important stuff (http://homepage.psy.utexas.edu/homepage/class/Psy391P/Josephs%20PDF%20files/Maxwell.PDF).

So why isn't there a "80% or 95% power"- badge to indicate a study's power and to indicate "good practices"?

Would that be useful?

I would love to hear your opinion...

[Brian Nosek]

It is a nice idea.  The main challenge is that in most cases, you can know the power of the design to detect any particular effect size, but you don't have more than best guesses for the effect size under investigation.  

In many of our projects, we use power analysis to help decide what sample size we need in order to be confident that we could detect an effect of the size that we care about.  But, there is no applicable standard for that. 

So, in practice, the badge would need to be specified as "80% power to detect an effect size of at least 0.5 standard deviations" so that both power and effect size are benchmarked.    

Brian

--
You received this message because you are subscribed to the Google Groups "Open Science Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openscienceframe...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[a.]

Thank you for the reply! I am still thinking about it.

Would there be another way in which you could incentivise high powered designs (as I understand is a useful thing to aim for) ?

Kind regards.

To unsubscribe from this group and stop receiving emails from it, send an email to openscienceframework+unsub...@googlegroups.com.

[Eric-Jan Wagenmakers]

Well, *before* you conduct a study it is great to have good power, and maybe people should be awarded for that. After running the study, however, power is less useful: high-power experiments can yield uninformative outcomes, and low-power experiments can yield very diagnostic outcomes. So I am hesitant about the badge, as it may give people the false idea that with high power the results are informative no matter what. Also, power is a frequentist concept and I am don't think this merits a badge. Finally, power depends on an assessment of expected effect size under H1. If I want this badge, all I need to do is greatly overestimate the effect size. Will be have to check the reasonableness of the power analysis before awarding the badge? So yes, high power is good and should be encouraged, but I see problems with the badge. I do recommend a badge "BF" for "authors reported Bayes factor instead of p value" :-)
EJ

--

You received this message because you are subscribed to the Google Groups "Open Science Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openscienceframe...@googlegroups.com.

[a.]

Thank you for your reply!

"Well, *before* you conduct a study it is great to have good power, and maybe people should be awarded for that"

So what would that look like in practice then: having good power before you run your study? Would you base your power calculation on the average attained effect size in psychological literature or something like that?

Do you perhaps have any other ideas on how to incentivise "good power"?

Kind regards,

Alexander

To unsubscribe from this group and stop receiving emails from it, send an email to openscienceframework+unsub...@googlegroups.com.

[Eric-Jan Wagenmakers]

Good question. If you do pick average effect size as a target (what is
this? d=.3?) then you can have 80% and 95% power badges which directly
translate to a specific number of observations or participants (these
will differ somewhat according to the experimental design). So
equivalently you could earn a badge for testing, say, at least 50
participants per condition in a between-subject design. In principle
I'm OK with this -- it might prompt people to test more participants
and get more reliable results.

My Bayesian problem with this is that I might specify a sequential
sampling plan along the lines of "I'll continue testing until the
Bayes factor is 10 or until I've tested at least 50
subjects/condition" -- you then obtain the desired level of evidence
after 40 subjects and consequently miss out on the badge.

On the other hand, it feel silly now to complain about such details.
We all agree that people should test more participants, so if badges
can nudge this process along that would be great. I withdraw my
earlier reservations, but I do think it is useful to construct
unambiguous guidelines for obtaining the badges.

Cheers,
E.J.
********************************************
Eric-Jan Wagenmakers
Department of Psychological Methods, room 2.09
University of Amsterdam
Weesperplein 4
1018 XA Amsterdam
The Netherlands

Web: ejwagenmakers.com
Book: bayesmodels.com
Stats: jasp-stats.org
Email: EJ.Wage...@gmail.com
Phone: (+31) 20 525 6420

“Man follows only phantoms.”
Pierre-Simon Laplace, last words
********************************************

>>> email to openscienceframe...@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Open Science Framework" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to openscienceframe...@googlegroups.com.

[a.]

Well, thank you all for the comments. I feel that I am a little disappointed in the lack of clear cut benchmarks or some practical rule of thumb. As I understand it power is very important, but to translate this importance to a practical rule of thumb seems hard to me. As I understand it, I could perform 1) a pilot study, or 2) look at relevant other literature to get an effect size estimate and then calculate what sample size is needed to achieve 80% power, but to me it seems that both 1 and 2 can lead to biased effect size estimates.

More importantly I wonder how to incentivise using highly powered designs beyond merely saying something like "use highly powered designs".