upcoming workshops on sensitivity analysis: Society for Research on Educational Effectiveness (Chicago, Oct 8); Association for Public Policy Analysis and Management (Seattle, Nov 14)
Ken Frank
Here are two upcoming workshops. Perhaps see you there. Please feel free to circulate.
We also recorded a short (7 minute) intro video here or watch the first hour of a course on the Statistical Horizons YouTube channel!
Ken
Society for Research on Educational Effectiveness (SREE)
October 8, 2025 (9:00am–1:00pm Central time). In person: Chicago, IL.
Description:
Understanding how research can be generated and interpreted across systems requires stakeholders in different systems and levels to communicate clearly about the strength of evidence supporting an inference. Towards that end, this workshop informs discussion about the strength of evidence by imagining the conditions necessary to change the inference. In part I, we use the potential outcomes and counterfactual framework to interpret how much bias there must be to nullify an inference. This supports statements such as “To nullify the inference, __% of the data would have to be replaced with counterfactual cases for which there was no effect.” In part II, we quantify the robustness of causal inferences in terms of correlations associated with unobserved variables. This supports statements such as “To nullify the inference an omitted variable would have to be correlated at __ with both the focal predictor and outcome.”
By leveraging intuition based on potential outcomes and correlations with omitted variables, these techniques inform broad stakeholder debate about use of evidence. Furthermore, because the techniques are functions of quantities commonly reported in published studies (e.g., estimated effect, standard error) they can be broadly applied and extended to a range of models (e.g., logistic regression, differences in differences, hazard models, propensity score based, mediation). Correspondingly, the techniques have already been widely employed in policy, education, sociology, epidemiology, and management, as well as in general science.
Calculations for bivariate and multivariate analysis will be presented in the app: https://konfound-project.shinyapps.io/konfound-it/ as well as modules in Stata and R. The format will be a mixture of presentation and individual hands-on exploration including application to studies selected by the participants. Participants will also have an opportunity to compare with alternative techniques (e.g., Oster’s Coefficient of Proportionality; Cinelli & Hazlett’s Robustness Index). This workshop is intended for anyone familiar with general quantitative methods including regression and analysis of variance as well as more sophisticated techniques (e.g., propensity scores, multilevel models).
Association for Public Policy Analysis and Management (APPAM)
November 14, 2025 (1:45–3:15pm Pacific time). In person: Seattle, WA.
Description:
The workshop informs discussion about the strength of evidence supporting a policy or practice by quantifying the conditions necessary to change an inference. In part I, we use the potential outcomes and counterfactual framework to interpret how much bias there must be to nullify an inference. This supports statements such as “To nullify the inference, X% of the data would have to be replaced with counterfactual cases for which there was no effect.” In part II, we quantify the robustness of causal inferences in terms of correlations associated with unobserved variables. This supports statements such as “To nullify the inference an omitted variable would have to be correlated at __ with both the focal predictor and outcome.” By leveraging intuition based on potential outcomes and correlations with omitted variables, these techniques inform collaborative evidence generation and evaluation among a broad set of stakeholders. Furthermore, because the techniques are functions of quantities commonly reported in published studies (e.g., estimated effect, standard error) they can be broadly applied and extended (e.g., logistic regression, hazard, propensity score based). Specifically, the techniques have already been widely employed in policy, education, sociology, epidemiology, and management, as well as in general science (e.g., Proceedings of the National Academy of Sciences).
We introduce participants to calculations using modules in Stata and R, as well as the app (https://konfound-project.shinyapps.io/konfound-it/). There will be a mixture of presentation and individual hands-on exploration. Participants will also have an opportunity to compare with alternative techniques (e.g., Oster’s Coefficient of Proportionality; Cinelli & Hazlett’s Robustness Index). This workshop is intended for anyone familiar with the general linear model but also informs more sophisticated techniques (e.g., propensity scores, logistic regression, multilevel models).