Articles and Other Useful Statistical Resources

[David Lovis-McMahon]

I believe developing statistical guidelines for the HCI community can be furthered by providing a resource for both authors and reviewers. To that end, I thought it might be good to get that ball rolling with a fresh epidemiology methods paper by Sander Greenland and colleagues (see attached) Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2015). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 1-14.

[Pierre Dragicevic]

Thanks David, this seems to be an excellent resource to cite in the guidelines.

I suppose it could be useful at some point to discuss what these guidelines should be trying to achieve. For now I see three possible goals.

1. Moving towards reviews that are more rigorous and less forgiving of errors in statistical analyses/interpretations. This would be desirable but as we all keep pointing out, we're lacking statistical expertise in the reviewing pool. So I don't think we should be  too obsessed with this goal. Educational material and resources that discuss common statistical errors are plentiful and we should encourage reviewers to read them, but I don't think the CHI guidelines necessarily need to repeat these.

2. Moving towards reviews that recognize and reward good practices that are not widely recognized as such at CHI. Things like clarity and completeness, nuanced conclusions, shared material, etc. are all easy to assess even by non-expert reviewers, and if reviewers are properly educated on the importance of these, this could contribute to improving the quality and transparency of reports overall and reduce practices like p-hacking.

3. Moving towards reviews that do not reject statistical reports for the wrong reasons. I'm not sure why this one is so much overlooked. For a non-expert and/or hurried reviewer, it is tempting to use simple heuristics to assess the validity of a statistical report (e.g., does it report ANOVAs / p-values? Are the results significant? Is sample size more than X?) rather than looking at the subtleties of the analysis or at the big picture. As long as reviewers will believe in such heuristics, other recommendations will have little influence.

My hope is that we can encourage reviewers to replace their old, misguided heuristics (3) with other, better heuristics (2) for evaluating studies. Covering (3) is difficult and we may not all agree, but it seems fairly easy to list the different ways we address concrete statistical problems, and state that they're all valid. Such a statement may seem vague as recommendation to authors, but as a recommendation to reviewers it is quite specific, because it implies that using method X rather Y shouldn't be a reason for rejection.

Pierre

On Sun, May 29, 2016 at 7:56 AM, David Lovis-McMahon <dlo...@gmail.com> wrote:

I believe developing statistical guidelines for the HCI community can be furthered by providing a resource for both authors and reviewers. To that end, I thought it might be good to get that ball rolling with a fresh epidemiology methods paper by Sander Greenland and colleagues (see attached) Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2015). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 1-14.

--
You received this message because you are subscribed to the Google Groups "Transparent Statistics in HCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to transparent-stat...@googlegroups.com.
To post to this group, send email to transparen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/transparent-stats-hci/a14e208a-c2bb-4961-bdff-3957740ad9c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.