Society for Research on Educational Effectiveness (Chicago, Oct 8) workshop NOW FREE; Association for Public Policy Analysis and Management (Seattle, Nov 14)
Ken Frank
NOW FREE! Society for Research on Educational Effectiveness (SREE) NOW FREE!
October 8, 2025 (9:00am–1:00pm Central time). In person: Chicago, IL.
Description:
Understanding how research can be generated and interpreted across systems requires stakeholders in different systems and levels to communicate clearly about the strength of evidence supporting an inference. Towards that end, this workshop informs discussion about the strength of evidence by imagining the conditions necessary to change the inference. In part I, we use the potential outcomes and counterfactual framework to interpret how much bias there must be to nullify an inference. This supports statements such as “To nullify the inference, __% of the data would have to be replaced with counterfactual cases for which there was no effect.” In part II, we quantify the robustness of causal inferences in terms of correlations associated with unobserved variables. This supports statements such as “To nullify the inference an omitted variable would have to be correlated at __ with both the focal predictor and outcome.”
By leveraging intuition based on potential outcomes and correlations with omitted variables, these techniques inform broad stakeholder debate about use of evidence. Furthermore, because the techniques are functions of quantities commonly reported in published studies (e.g., estimated effect, standard error) they can be broadly applied and extended to a range of models (e.g., logistic regression, differences in differences, hazard models, propensity score based, mediation). Correspondingly, the techniques have already been widely employed in policy, education, sociology, epidemiology, and management, as well as in general science.
Calculations for bivariate and multivariate analysis will be presented in the app: https://konfound-project.shinyapps.io/konfound-it/ as well as modules in Stata and R. The format will be a mixture of presentation and individual hands-on exploration including application to studies selected by the participants. Participants will also have an opportunity to compare with alternative techniques (e.g., Oster’s Coefficient of Proportionality; Cinelli & Hazlett’s Robustness Index). This workshop is intended for anyone familiar with general quantitative methods including regression and analysis of variance as well as more sophisticated techniques (e.g., propensity scores, multilevel models).