Major flaws in the Lohse et al. review of MBI

[Will Hopkins]

Janet Aisbett has independently critiqued the recent review of the use of magnitude-based inference (MBI) in more than 200 publications by Keith Lohse, Kristin Sainani and others. Janet's critique is published as a comment following the Lohse et al. review in PLoS One.

In summary, Lohse et al. had correctly identified that the majority of MBI authors had used non-clinical MBI for clinically important outcomes. However, Lohse et al. also claimed that the MBI authors had effectively presented possibly or likely beneficial or substantial outcomes as decisively, conclusively or definitely substantial and thereby incurred the high Type-1 error rates that Kristin Sainani has always claimed for MBI. Janet Aisbett's careful reading of a sample of the MBI publications shows that the authors did NOT make such conclusions to anywhere near the extent claimed. There are major methodological flaws in the review, which Janet has detailed in her critique.

Conclusions: The use of non-clinical MBI for clinically or practically important outcomes is a minor error, because the authors did not recommend implementation of a treatment or claim that the outcome was anything other than possibly or likely beneficial. This kind of error is easily fixed with more vigilance by authors, reviewers and editors. The Lohse et al. review has damaged the literature by using flawed methodology to make unsubstantiated claims about MBI, which is in fact a valid inferential method that does not suffer from high error rates and that correctly presents sampling uncertainty in an accessible and practical manner. See my first message in this forum for more.

Will