A Simple Metric for a Complex Outcome: Proposing a Sustainment Index for Health Indicators

"Ultimately, quantifying the degree of sustainment of program investments only makes sense if research is sought to guide policy and program investments."

Sustainability is one criterion for evaluating development and health programmes. In this context, sustainability refers to: what happens to communities and beneficiaries after external funding ends; institutionalisation of practices within local social or institutional systems; or maintenance of an achieved outcome. In this paper, Eric Sarriot and Reeti Desai Hobson propose a simple metric, free of causal inference, which can be used to test different empirical models for the "sustainment" - that which can be observed looking back in time - of health outcomes.

The emphasis in the sustainment index (hereafter, SI) is on the measured sustainment of health outcome indicators, or acceptable proxies, such as immunisation coverage. While the authors believe that sustainability is non-linear, highly contextual, and process oriented, in the context of this focus on health outcomes, the questions are: How much was a benefit sustained, and how did this happen? The hope is that proposing a new metric as a dependent variable will stimulate new research inclusive of post-project periods and achievement in their designs. But for now, it would be interesting for researchers who have already collected baseline, endline, and post-project service or outcomes data to plot how the SI performed, if just to observe its performance.

The SI is a quantification of a trend change over 2 periods for a health indicator - it carries no assumptions about human agency, programme contribution, attribution, secular trend, or other. So, the index is not a measure of absolute good, but relative inflection of a trend between T0 and T1. It is insensitive to the actual values of the indicator. Progress on coverage from 20% to 40% to 60% would yield the same exact SI (2.0) as progress from 20% to 25% to 30%. In other words, the SI measures the level of sustainment of a performance (how the slope of progress is sustained), not the performance of the indicator (the inclination of the trend).

Sarriot and Hobson present the construction of the SI for health indicators, as the dependent variable against which different models can ultimately be tested. Operationally, the goal is to measure how a health indicator (Y) trend from time T0 to T1 continued from time T1 to T2 (presumably after a transition phase / end of a project). The SI is a simple-to-calculate approximation of the derivative of Y over time (T0: baseline, T1: endline, and T2: post-project), based on the ratio of the slope of Y T1-T2 over Y T0-T1. The construct provides 3 clear benchmarks: SI = 0, when the health indicator returns to baseline value post-project (Y T2 = Y T0); SI = 1, when the endline-post-project trend is a plateau; and SI = 2, when the progress slope during the programme is uninterrupted post-programme.

Sarriot and Hobson tested the validity of the SI metric through 3 analytical steps: a face validity validation exercise; application to a real post-project dataset; and application to Demographic and Health Surveys (DHS). For example, the SI was used in the following way: 16 countries out of 22 (with at least 3 waves of DHS surveys conducted between 1994 and 2014) had results that allowed for computation of a SI for full vaccination coverage. One showed limited sustainment (Mali), 5 had near-plateau sustainment (Guinea, Namibia, Côte d'Ivoire, Mozambique, Nigeria), 5 indicated strong sustainment (Cameroon, Ethiopia, Uganda, Madagascar, Senegal), and 5 showed accelerated progress (Ghana, Burkina, Tanzania, Rwanda, Niger).

Sarriot and Hobson found strong correlation (r2 = 0.922) between the SI and independent practitioners' rating of indicator trends. "In effect, our index shows potential for helping us ask why different indicators behaved differently in a post-project period, and helps us discriminate between relatively higher or lower levels of sustainment."

In concluding, Sarriot and Hobson explain that the SI "allows comparisons of the behavior of different indicators in the same context, as well as the comparison of the behavior of the same indicator in different contexts - something evidently more useful if the researcher knows something about the contexts and factors at play. Its merit is in providing a quantification of an outcome, which appears desirable - continuation of progress under changing conditions - and in allowing comparison within sensible and empirical parameters....[W]e hope that this simple metric can help abolish the question 'is it sustainable?' in favor of...empirically-testable ones, such as 'how much was sustained?' and 'how?'"