Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment

"From those just beginning their digital journey to more sophisticated project teams, we hope that this Toolkit will continue to advance the quality of monitoring and evaluation activities within digital health and contribute to more evidence-based planning, implementation and investment."

This resource provides step-wise guidance to improve the quality and value of monitoring and evaluation (M&E) efforts in the context of digital health interventions, also commonly referred to as mHealth or eHealth interventions. Developed by the World Health Organization (WHO)'s Department of Reproductive Health and Research, the United Nations (UN) Foundation, and the Johns Hopkins University Global mHealth Initiative, the guide is intended for implementers and researchers of digital health activities, as well as policymakers seeking to understand the various stages and opportunities for systematically monitoring implementation fidelity and for evaluating the impact of digital health interventions.

The resource represents the collective learning from 5 years of engagement with 26 agencies working to strengthen their digital health deployments, develop robust evaluations, and scale up their activities nationally and regionally. The lessons learned from working with these partners are described in this document, which provides high-level guidance and systematic direction to programme planners and implementers embarking on similar journeys. Grounded in the real experiences that have emerged from numerous projects across 3 continents, this tool offers guidance ranging from development of an M&E plan to monitoring implementations, evaluating outcomes, assessing data quality, and eventually, to reporting evaluation findings using an EQUATOR Network-endorsed checklist. Furthermore, this document provides the tools necessary to ensure that evidence from digital interventions is of sufficient quality to inform the upcoming WHO Guideline on Digital Health Interventions for Health System Strengthening document, featuring recommendations to Ministries of Health and their implementing partners.

Specifically, after introductory sections, Chapter 1 begins with a broad overview, describing the goals for M&E and explicitly distinguishing between the efforts aimed at monitoring implementations and those aimed at evaluating their impact. It explores developing an M&E plan for your digital health intervention. Chapter 2 guides the reader to formulate specific intervention claims (statements of anticipated benefits of the digital health system or intervention) and develop indicators specific to those claims, including the selection of process indicators that reflect implementation fidelity. This chapter also introduces readers to the selection and development of a framework to guide the intervention assessment. Chapter 3 then takes readers through the set-up of a monitoring plan, focusing on technical stability and performance. It includes discussion of tools for monitoring and digital health process monitoring components. Chapter 4 shifts to the realm of evaluation, introducing the reader to qualitative, quantitative, and economic methods commonly used to generate data in support of programme claims. It seeks to help answer the question: Which evaluation activities are right for you? Chapter 5 focuses on methods for assessing, and improving, the quality of data being collected. It introduces the data quality assessment approach, featuring a data quality assessment worksheet and instructions and a sample application. The last part of the guide, Chapter 6, focuses on reporting findings from the programme - described here as an often neglected but critical area, because decision-makers look to these findings for support when seeking to invest in digital health strategies. Presented here is the mHealth Evidence Reporting and Assessment (mERA) checklist, with instructions on how to use mERA and an examination of methodological criteria. Annex I is a glossary.

Although more in-depth texts and curricula are available on the methods discussed, this guide focuses on presenting pragmatic highlights and experience-informed tips for implementers to consider, together with text boxes with case studies, charts, and links and resources for further study.