Misuse of Statistical Significance and Non-significance

[Will Hopkins]

Here's a draft abstract of an article I am trying to find time to write arising from my report in Sportscience on last year's virtual ECSS conference. It should be useful for anyone intending to present sample-based research at this year's conference (deadline April 30) or in any journal. Please get back to me with any helpful criticism or encouragement. Thanks.

Will

Misleading Conclusions based on Statistical Significance and Non-significance at a Recent International Conference

Will G Hopkins, Victoria University, Melbourne, Australia

Rationale. Conclusions about effect magnitudes based on statistical significance and non-significance may sometimes be misleading, since rejection or failure to reject the nil hypothesis does not necessarily imply that an effect is substantial or trivial respectively. I have therefore assessed such conclusions in a sample of 32 studies relevant to athletes at the 2020 annual meeting of the European College of Sport Science.

Methods. I used the criterion evidence for substantial and trivial effects provided respectively by rejection of non-substantial and substantial hypotheses; I also used magnitude-based decisions (MBD), which incorporates such hypothesis testing. I chose smallest important magnitudes and alphas (p-value thresholds) for the hypothesis tests and for MBD that are appropriate for effects on injury, fitness and performance. I assessed all significant and non-significant effects presented with sufficient data to estimate sampling uncertainty as p values via standard errors
Results. Only 18 of 32 significant effects were decisively substantial (rejection of a non-substantial hypothesis, pN+<0.05 or pN--<0.05), while three were actually decisively trivial (rejection of both substantial hypotheses, p+<0.05 and p--<0.05). Of 19 non-significant effects, none was decisively trivial. MBD provided usefully nuanced probabilistic assessments of magnitude of significant and non-significant effects, especially for those clinically relevant effects where the hypothesis of harm but not of benefit was rejected (pH<0.005, pB>0.25). 

Conclusion. Assessments of magnitude based on statistical significance and non-significance were grossly misleading at this conference. Researchers should account for sampling uncertainty by replacing the nil-hypothesis test with tests of substantial and non-substantial magnitudes or by using MBD.