A Mixed-Methods Evaluation to Determine the Effects of a Novel mHealth Platform for Maternal Child Health Tracking in Rural Udaipur, India

"Altogether, the findings...represent strong evidence from a large, randomized, prospective trial, that even without the full scope of the Khushi Baby application in play and despite challenges, there were significant improvements in data quality, health behaviour outcomes, health outcomes supported by health worker and health official satisfaction."

According to India's 2015-16 National Family Health Survey, less than half of the children in Rajasthan's Udaipur district received all of their vaccinations. In 2014, rural Udaipur's full infant immunisation rate was estimated to be even lower, at 37.2%. Going beyond increasing coverage, improving timeliness of immunisations is critical to infant health in the first year of life. Carried out with funding from the International Initiative For Impact Evaluation (3ie), this impact evaluation tests a combination of several components for community engagement, including but not limited to the wearable, culturally-symbolic, near-field communication (NFC)-based Khushi Baby (KB) pendant and dialect-specific voice calls for maternal and child health (MCH) education against the status quo control.

In response to the challenges of paper-based tracking at the field level, several solutions have been developed to digitally collect MCH data in rural healthcare settings. However, they are vulnerable to limitations, such as issues with respect to data accountability - e.g., health workers may still create records offline without physically seeing beneficiaries. In contrast, the KB pendant stores the medical history of the beneficiary, allowing them to carry their updated health record to any health worker in a digital format. The health worker cannot update or create a new health record without the beneficiary's biometric and KB pendant being scanned to match. The KB platform also was designed to make culturally-informed improvements to a technological approach, through community-inspired design of the wearable digital health record and with patient-specific, dialect-specific voice reminders for populations that are largely illiterate. (For more on KB, see Related Summaries, below.)

Specifically, the intervention aimed to better connect health workers with beneficiaries through data-driven engagements, using:

• Digitally encrypted necklaces for expecting mothers and infant children and biometric-enabled tablets: Keeping the socio-cultural norms in mind, these necklaces were designed like an amulet ("taveez" in Hindi) strung on a black thread, which in some cultures is symbolic for having protective powers. The necklaces contained an unpowered digital micro-chip storing all the health data of registered mothers and infants. This chip was read and updated by a frontline health worker, known as the Auxiliary Nurse Midwife (ANM), using an android tablet at a health camp.
• KB application (app) and dashboard: AMNs used an Android app, made for tablets. The data recorded on the micro-chips and tablets with biometric identification features were synced on KB's mHealth platform, which can be accessed through a web-based dashboard.
• Voice call reminders to caregivers and health workers: Automated and personal voice call reminders about antenatal care (ANC) visits or immunisation camps were sent to all expecting mothers and caregivers registered in the KB system.
• KB field monitors and Whatsapp groups: Local KB staff added health workers and their supervisors (Block Medical Officers, or BMOs) to WhatsApp groups. On a weekly basis, the monitors shared high-risk patient reports with the ANMs in the groups. The groups were also used to share educational content related to specific MCH themes. ANMs used the groups to report back on high-risk patients from their catchment areas.

The study was a cluster-randomised controlled trial conducted in Udaipur. A cluster was defined as the catchment area of a sub-centre, which is the rural health outpost in the Indian health system. A total of 100 sub-centres were included in the study and were equally allocated to treatment and control. Approximately 1,300 caregivers were interviewed at baseline and endline to assess the impact of the intervention on immunisation coverage.

The implementation uptake from February 2017 to June 2018 (the evaluation period) showed that 87 ANMs used the KB app, although significant challenges were faced - e.g., technical issues and other weekly crashes within the app directly caused many ANMs to express frustration and desire to abandon it. Although trainings were provided, other health workers and officials did not consistently use the KB dashboard over the evaluation period. In short, most engagement came from automated voice call reminders through the KB dashboard, WhatsApp group messages exchanged by heatlh worker teams, and personalised voice calls to high-risk and drop-out mothers. For example, on a weekly basis, over 20 WhatsApp messages are exchanged and over 1,100 voice call reminders are automatically sent to mothers on average.

Key findings:

• The unadjusted mean proportion of infants fully immunised per recall was 72.7% in the treatment arm and 57.7% in the control at endline, representing a respective increase from baseline of 47.3 percentage points and 35.0 percentage points. The unadjusted difference-in-difference in full infant immunisation was 12.2 percentage points higher in the treatment arm, which was both programmatically meaningful and statistically significant.
• KB was found to have improved data completeness and data consistency by nearly 20 percentage points, and it reduced the time to acquire data to a median of just under 4 hours.
• After adjusting for confounders and effects of clustering, mothers randomised to sub-centres that received the KB intervention were 1.66 times more likely to report full infant immunisation at the endline than those mothers randomised to control sub-centres.
• Mothers in treatment arm were more likely to report higher awareness about maternal immunisations than the control.
• The number of voice calls received, socioeconomic status (SES) score, geographical block, ASHA visit frequency, higher MCH awareness (higher iron-folic acid (IFA) consumption, delivery at hospitals, and exclusive breastfeeding), ANM use of mobile phones, and ANM proximity to primary health centres (PHCs) were all found to be significant factors in multiple outcome models.
• Focus group discussions with mothers generated feedback around 3 thematic areas:
  1. While some mothers appreciated the voice call messages, others had perceptions that the call frequency was inconsistent or easily mistaken for an advertisement.
  2. Some mothers found the pendant to be convenient and customisable, while others felt that the reason for why the pendant needed to be worn was not properly explained. Still others found that the whole KB process as a result of the pendant scan may have increased the time spent at the camp.
  3. Ideas for improvement included guidance on foods to eat during pregnancy, ability to customise the pendant, and access to more educational content with local language videos.
• Testimonies from the ANMs in the treatment arm indicate they were satisfied with the intervention and perceived it to be the cause of improved beneficiary attendance at health camps.
• Health officials and supervisors said they appreciated the system, particularly the WhatsApp groups, through which they reported that they saw an improvement in how the ANM was addressing high risk and drop out beneficiaries. There were calls to have the system expanded and integrated into the Rajasthan State Ministry of Health's Pregnancy, Child Tracking & Health Services (PCTS) database.

The findings indicate that a system with demand and supply components driven by digital data collection, distribution, and weekly follow-up may improve immunisation outcomes in similar rural areas, particularly those that share cultural beliefs around using amulets for child protection.

The authors conclude that the lessons from this evaluation could help policymakers understand the feasibility, acceptability and effectiveness of mhealth systems and whether these hold the potential to replace the current paper-based health tracking systems. In fact, engagement on the KB platform and evaluation findings informed the Rajasthan government's decision to engage the KB team as a nodal technical partner for the Nirogi Rajasthan Scheme, under which frontline workers will conduct a health census, collecting information from across the state using a customised mobile app.