brownbag: Quantifying Clinical Uncertainty in times of Crisis: How Different Would a Sample have to be to Change an Inference? Oct 9
Ken Frank
Quantifying Clinical Uncertainty in times of Crisis: How Different Would a Sample have to be to Change an Inference?
Abstract
Early evidence on the efficacy of a treatment often comes from single studies with a high degree of uncertainty. This is true even for well-designed and executed randomized controlled trials, as control and treatment groups can be imbalanced, unintentionally by an experimenter’s action or simply by chance. Unfortunately, conventional methods for expressing that uncertainty -- standard errors and confidence intervals -- are statistical constructs notoriously prone to misinterpretation. This problem is amplified by the COVID-19 global pandemic, where it is crucial that a broad set of stakeholders have a common understanding of the strength of the inferences drawn from emerging research. In this paper, we present an approach for expressing the robustness of study inferences in terms of hypothetical changes to the underlying data. This generates statements such as “The inference would change if xx of the treatment patients who experienced a benefit were replaced by patients for whom there was no effect of the treatment.” This characterizes the confidence of an inference in relatable terms that presume little statistical knowledge and, similar to the concept of fragility, can be particularly helpful in identifying studies where statistically significant results might not be particularly robust.
Key words: causal inference; fragility; replacement of cases; meta-analysis
please join if you like.
https://msu.zoom.us/j/94505246831 (Password: Brownbag)
Check out the R Shiny app for sensitivity analysis
http://konfound-it.com or the Blog https://www.konfound-it.org/
Check out our Causal inference and COVID working paper:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3607967
Ken