Analyzing the Usage of Theories of Change for Routine Immunization Programs - A Review of Impact Evaluations from LMICs

"...[This] review highlights the strong need to strengthen theory-based evaluation approaches for the routine immunization of children in LMICs."

Since the early 1990s, theories of change (TOCs) have been used as an implementation tool to aid programme development, internal organisational communication, external dissemination and advocacy, programme adaptation and learning, and, more importantly, to determine how to evaluate programme implementation and its effectiveness. This article analyses the extent of the usage of TOCs and causal pathways in the evaluation of immunisation programmes to identify the challenges to generating evidence on how interventions improve immunisation.

The researchers conducted this review of impact evaluations (IEs) published after 2010 from a representative sample of countries from low- and middle-income countries (LMICs). They searched for studies from the Evidence Gap Map (EGM) on routine childhood immunisation and a review conducted by the Yale Institute for Global Health (both available at Related Summaries, below). The last searches of EGM and Yale review were conducted in May and March of 2020, respectively.

The review included 47 large-scale and 45 small-to medium-scale interventions. Of the included studies, 19% used a TOC, 56% described a causal pathway or used a conceptual diagram with varying degrees of detail, and 25% of the IEs did not provide any information on how their intervention was expected to affect change. Only 19 of the 92 IEs explicitly outlined any assumptions associated with the implementation of the interventions. Forty studies measured the outputs or intermediate outcomes leading to improved immunisation coverage. Of the programmes with a TOC, 11 IEs measured the outputs and intermediate outcomes that could affect immunisation uptake. For example, Oyo-Ita et al., in their evaluation of the programme on engaging with traditional religious leaders (TRLs), measured: the degree of community engagement; qualitative assessment of knowledge, attitudes, and beliefs; and satisfaction with TRLs. Of those without a TOC but with a causal pathway, 13 had one or more intermediate outcomes. For example, interventions that focused on training or educating mothers about child health practices using face-to-face interactions or m-health interventions measured the knowledge levels among mothers.

The review has the following policy and research implications for immunization programs:
 

1. Evidence on how large-scale health system strengthening (HSS) programmes affect immunisation has limited use for scalability without a detailed TOC. In the context of immunisation, using TOCs to understand how health system changes influence the wider system has greater importance, as routine immunisation services are integrated with other child and development services and can be influenced by other supplementary vaccination programmes for emerging infections such as COVID-19. They can also depend equally on the demand side of the intervention.
2. Integration of TOC into evaluation needs to use multiple methods and approaches. Multicomponent interventions that operate at the familial, community, and health system levels did not measure many causal links between the intervention and immunisation outcomes. Per the researchers, there is a need to use mixed methods and novel approaches to better understand how interventions cause change. Future research needs to leverage multiple approaches and use both qualitative and quantitative methods for evaluation.
3. Evaluations need to be based on programme theories. The lack of a TOC and any relevant theory informing the intervention strategy adds to the gap in our understanding of how programmes are expected to make an impact on routine immunisation. It also prevents the synthesis of evidence on what worked and how in the context of existing theories.
4. Context needs to be accounted in the design and evaluation of programmes. The review showed that context is mostly used to provide only a background or rationale for the intervention and to explain the challenges in the implementation or evaluation of the programmes in a few studies. There is a need to move away from looking at interventions as separate packages of components that are introduced in a certain context. In developing and justifying a TOC to inform an intervention and its evaluation, researchers should show a clear understanding of how the context influences the programme and vice versa. An intervention that is effective in some settings could be ineffective or even harmful elsewhere.

In conclusion: "Future implementers and evaluators need to develop clear TOCs that are based on established theory and have clearly articulated the underlying assumptions. Importantly, for the evidence to be usable for policy or practice, the lack of evidence on causal mechanisms can be a major roadblock. An increase in complex programs also require novel approaches and the use of multiple evaluation methods....There is a need to combine both impact and process evaluations and to use implementation research techniques to understand how the intervention has affected the outputs and outcomes that can impact immunization uptake."