Outcomes Intermediate

[Kiliano Ratha]

Onedifference could be the length of time between the program intervention and the measured outcome, however the most important difference is the effect the intervention has on the outcome. Short-term outcomes can be directly tied to the intervention, while long-term outcomes can be less directly attributed to the program. In general, short-term outcomes are measured at the end of the program or soon after the program has finished. Short-term outcomes refer to changes in knowledge, attitudes, or behaviors and can include reports of behaviors that participants intend to change or motivation to change. Intermediate outcomes are usually measured within several months after the end of the program and include actions by participants based on what they learned. Long-term outcomes are measured a year or several years after program completion and include changes in conditions, policies, or organizational structure. For example, a short-term outcome for a smoking prevention program for teenagers could be the number of teens who report that they do not plan to start smoking. An intermediate outcome could be the number of teens who report not smoking at six months, and a long-term outcome could be a reduction in the smoking rate among teens in a city, county, state or region. Short-term, intermediate, and long-term outcomes are related and build on each other.

This work is supported in part by New Technologies for Agriculture Extension grant no. 2020-41595-30123 from the USDA National Institute of Food and Agriculture. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the U.S. Department of Agriculture.

The site is secure.

The ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Importance: As many as 10% of women experience natural menopause by the age of 45 years. If confirmed, an increased risk of cardiovascular disease (CVD) and all-cause mortality associated with premature and early-onset menopause could be an important factor affecting risk of disease and mortality among middle-aged and older women.

Objective: To systematically review and meta-analyze studies evaluating the effect of age at onset of menopause and duration since onset of menopause on intermediate CVD end points, CVD outcomes, and all-cause mortality.

Study selection: Studies (ie, observational cohort, case-control, or cross-sectional) that assessed age at onset of menopause and/or time since onset of menopause as exposures as well as risk of cardiovascular outcomes and intermediate CVD end points in perimenopausal, menopausal, or postmenopausal women.

Data extraction and synthesis: Studies were sought if they were observational cohort, case-control, or cross-sectional studies; reported on age at onset of menopause and/or time since onset of menopause as exposures; and assessed associations with risk of CVD-related outcomes, all-cause mortality, or intermediate CVD end points. Data were extracted by 2 independent reviewers using a predesigned data collection form. The inverse-variance weighted method was used to combine relative risks to produce a pooled relative risk using random-effects models to allow for between-study heterogeneity.

Main outcomes and measures: Cardiovascular disease outcomes (ie, composite CVD, fatal and nonfatal coronary heart disease [CHD], and overall stroke and stroke mortality), CVD mortality, all-cause mortality, and intermediate CVD end points.

If data are collected about these intermediate outcomes, and they can be linked to final impacts, it is possible to check whether all cases that achieved the final impacts achieved the intermediate outcomes.

In conjunction with options that address other tasks related to understanding causes (analyze counterfactual, and investigate exceptions) this provides a stronger causal analysis than simply reporting totals for outcomes and impacts.

Orange-colored varieties of sweet potato have been bred to increase vitamin A intake, particularly among children in Latin America. After the introduction of these varieties to a district, and an advertising campaign to explain their value, they were sold in local markets. The quantity of orange sweet potatoes sold was measured, as was the improvement in terms of reduced numbers of cases of Vitamin A deficiency. The evaluation would be strengthened if it were possible to collect and analyze data to check whether those children with improved health were in families that had bought the sweet potatoes and fed them to the children.

What is important here is both measuring intermediate outcomes (families feeding sweet potatoes to children) and also being able to link it to impacts in a crosstabulation. This provides better information than simply being able to report, for example, that 60% of families fed the sweet potato to their children and 60% of children showed clinical improvements in their level of vitamin A.

This table shows results that are consistent with a cause-effect relationship. All the children where families fed them sweet potato improved their vitamin A levels, and none of the other children did.

The following table shows results that are largely consistent with a cause-effect relationship. Most of the children where families fed them sweet potato improved their vitamin A levels, and few of the other children did. In this scenario it would be important to follow up these exceptions

The following table shows results that would not be consistent with a cause-effect relationship. There appears no relationship. Children not fed sweet potato were as likely to improve as those who were. (50% of both groups improved). In this scenario it would be important to investigate what else families were doing that might be increasing vitamin A levels outside the program.

The following table shows results that would not be consistent with a cause-effect relationship. Even where intermediate outcomes were achieved (feeding the sweet potato), final impacts were achieved by very few. In this scenario it would be important to use other methods to explore possible explanations.

It would also be important to Investigate exceptions by gathering more detail if possible about the families which achieved the final impacts, including checking dose-response - seeing if they were feeding more sweet potato than other families.

This type of pattern is sometimes seen when people adjust their behaviour in ways that undercut the way the intervention is supposed to work - for example, if after medical make circumcision men engage in riskier sex than previously. This can balance out any gains from the intervention.

The following table shows results that would not be consistent with a cause-effect relationship where the intervention contributes to producing the impacts. In this case it shows a pattern where those engaging in the intervention have worse outcomes than those who do not. In this scenario, children who were not fed sweet potato were more likely to improve their vitamin A levels. This is the reverse of the results that the logic model would predict and some investigation would be needed to explore possible explanations.

BetterEvaluation is part of the Global Evaluation Initiative, a global network of organizations and experts supporting country governments to strengthen monitoring, evaluation, and the use of evidence in their countries. The GEI focuses support on efforts that are country-owned and aligned with local needs, goals and perspectives.

Founded in 1974, MDRC is committed to improving the lives of people with low incomes. We design promising new interventions, evaluate existing programs, and provide technical assistance to build better programs.

An earlier post in this series discussed considerations for reporting and interpreting cross-site impact variation and for designing studies to investigate such cross-site variation. This post discusses how those ideas were applied to address two broad questions in the Mother and Infant Home Visiting Program Evaluation (MIHOPE): (1) whether variation in the way program services were implemented was related to variation in impacts across local programs, and (2) whether variation in the amount of services participants received was related to variation in impacts. Results from those analyses can be found in the MIHOPE 15-month impact report.[1]

The research team saw value in each of these approaches but also had concerns about the validity of each approach for the reasons outlined above. Meta-regression, instrumental variables, and causal mediation analysis were chosen for several reasons. Each could simultaneously look at several continuous, explanatory factors (such as aspects of program implementation for the meta-regression and types of home visiting services for instrumental variables and causal mediation analysis). The team also considered using two methods that rely on predicting who would be in a subgroup defined by a mediator, such as who would have received program services. The team decided not to use these methods in the published report because they were designed to look at a discrete number of subgroups.[6] In addition, instrumental variable and causal mediation analysis provided alternative ways of examining the relationship between program services and impacts, and the consistency of the results of the two analyses was one criterion used in deciding how much to emphasize specific findings.

[1]Charles Michalopoulos, Kristen Faucetta, Carolyn J. Hill, Ximena A. Portilla, Lori Burrell, Helen Lee, Anne Duggan, and Virginia Knox, Impacts on Family Outcomes of Evidence-Based Early Childhood Home Visiting: Results from the Mother and Infant Home Visiting Program Evaluation, OPRE Report 2019-07 (Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2019).

3a8082e126